According to a study by the International Data Group (IDG), unstructured data is growing at an alarming rate of 62% per year. The same study also suggests that by 2022, close to 93% of all data in the digital world will be unstructured!!

These statistics can be alarming for enterprises that are already grappling with the issue of having to deal with loads of unstructured data. There is a need for technology that can easily process this unstructured data and help organizations discover what's within it - with speed and accuracy - and text analysis is the answer.

Text analysis, often used synonymously with text mining, is the process of analyzing chunk of unstructured data to find out previously undiscovered information and insights that can be leveraged for informed decision making and other processes. The new age text analysis tools like 3RDi Search offer a host of text mining services like sentiment analysis, content classification, semantic search, content summarization, named entity recognition and more. Text analysis tools are based on a complex process that consists of several concepts, such as statistics, machine learning, natural language processing and more. It also involves the use of many techniques and this article talks about 5 of the most common ones.

Information Extraction:
Objective: Reconstructing a set of unstructured or semi-structured textual documents into a structured database.

• The first step in the process of evaluation of unstructured data.
• Involves tokenization and identification of named entities, key phrases and parts-of-speech.
• Uses concept of pattern matching to find out predefined sequences if any within the data.
• Identifies the relationship between entities and attributes.

Categorization:
Objective: Assigning one or more categories to an unstructured text document.

• Works on an input-output principle wherein the system is given inputs regarding the pre-defined categories under which the data in the new documents is to be classified.
• Consists of the following steps - processing, indexing, dimensional reduction and classification.
• Uses the Nearest Neighbour classifier, Decision Tree, Naïve Bayesian classifier, and other statistical classification techniques.

Clustering:
Objective: Bringing together clusters of documents that have similar content.

• Generates multiple groups of documents known as clusters.
• The content of documents in a specific cluster are very similar while that of documents in different clusters are not even remotely similar.
• Differs from clustering as it brings together documents without the use of any pre-defined categories as reference. This technique works on semantics - the principle on which semantic search engines work.
• K-means is a frequently used algorithm that brings great results.

Visualization:
Objective: Simplifying and enhancing the discovery of useful information with visual cues.

• Uses visual cues such as text flags to indicate individual documents or document categories and colours to indicate the density of a category, entity, phrase, etc.
• Enables the user to zoom in/out or scale the document as required, without any loss of data.
• Places large sources of textual data into a visual hierarchy.

Summarization:
Objective: Automatically generating a summary/compressed version of the text with information that will be of the highest importance or relevance to the end user.

• Determines the most important points in a lengthy document that the user of the text analysis tool will find useful.
• Involves 3 steps - Pre-processing, Processing, and Development.
• The pre-processing step involves building a structured representation of the text.
• The processing step involves application of algorithms to generate a summary of the text.
• Uses semantics technology, similar to a semantic search engine, to retain the meaning of the text in the summary.
• The development step is where the final text summary is obtained.

Information Extraction:

The techniques discussed above together contribute towards the efficiency of a text analysis tool. The new age text analysis tools have emerged as the must-have tools for enterprises in order to gain insights for informed decision making. With the rapid development in artificial intelligence and related concepts, the future holds unlimited possibilities for data processing and analysis with semantic search engines and text analysis tools alike.