What is Deep Text Analytics?

Deep Text Analytics (DTA) is a method for extracting information from unstructured data. It combines machine learning and NLP techniques with knowledge graphs to enable algorithms to better analyze text by not only processing words, but understanding the underlying concepts and their context.

The Challenge Posed by Text

Human language ability is unique. It is at the heart of our social and business interactions. Language enables us to communicate, cooperate, negotiate and interact with each other. It is the tool we use to capture our experiences and share our knowledge. Our laws, regulations, messages, disputes and business transactions are all documented in text form.

The amount of text produced by humanity is growing exponentially, and much of it is valuable business information. Therefore, the ability to process large amounts of text, gain insight from it, organize it, connect it, understand it, and use it to answer questions is of paramount importance. 

This is where our approach to deep text analysis comes in. Using semantic knowledge graphs, natural language processing and machine learning, we can effectively extract and classify information from large amounts of data. This method overcomes many of the limitations of other approaches that have similar goals but remain imprecise due to a lack of text context processing capabilities.

The Limitations of Text Mining 

You have probably heard of text mining, the process of extracting information from text using natural language processing (NLP) techniques. With text mining, typically large collections of documents are processed to discover new and relevant information or to help answer specific questions. The aim is to identify, extract and structure facts, relationships and assertions so that they can be further processed as text data by machines.

The structured data generated by text mining can be reused in semantic data fabrics, data catalogs or as business intelligence dashboards for data analysis.

Missing context: However, most text mining methods are primarily based on statistical procedures that lack any background knowledge that could be conveyed by ontologies, taxonomies or knowledge graphs. In simple terms, this means that the system searches for words in texts and counts how often they occur, how relevant the words are therefore (e.g., by calculating the so-called TFIDF), and how these words occur in the neighborhood of other words. This approach has many limitations, the largest of which is that machines cannot understand the semantic context in which the words are embedded.

Ambiguity: This leads to major problems when it comes to disambiguation, for example. If you, as a human being, read a text about cars and see the word “Jaguar”, you know that it refers to a car brand, because you understand from the context that “Jaguar” in this case must be the car and not the animal. 

Lack of standards: Another disadvantage of traditional text mining tools is that the resulting, often more structured data objects are not based on any standard and therefore cannot be easily processed together with other data streams, e.g., to be linked and matched with structured data. 

Such limitations make clear that we have to teach our AI applications how to understand words and context. That is the only way for them to determine their meaning more accurately.

Deep Text Analytics Goes Beyond Simple Text Mining

Deep Text Analytics uses knowledge graphs and semantic standards to process the context of the text to be analyzed, which can then be embedded in an even broader context. It is a very advanced methodology for automatic text understanding, based on a number of technologies that are being fused together: 

• Text structure analysis
• Extraction of entities from text based on knowledge graphs
• Extraction of terms and phrases based on text corpus statistics
• NLP techniques such as stemming or lemmatization
• Recognition of named entities and text classification based on machine learning enhanced by semantic knowledge models
• Optionally also the extraction of facts from text
• Automated sense extraction of whole sentences, which is based on the extraction of data and entities and validation against a set of conditions using knowledge graphs

This gives machines access to the relevant background knowledge to interpret and classify words, sentences and even entire paragraphs more precisely. This knowledge is made available as a knowledge graph based on W3C standards to resolve language problems of natural language more precisely. This approach helps to avoid misinterpretations, a common problem e.g. with virtual assistants who often take information at face value and do not have the ability to read between the lines.

All Benefits At a Glance

Here is a list of the advantages that Deep Text Analytics brings compared to traditional text analysis methods: