NLP & NLU Fundamentals

There are three different levels of linguistic analysis done before performing NLP:

1. Syntax: What part of the given text is grammatically right?
2. Semantics: What is the meaning of the given text?
3. Pragmatics: What is the purpose of the text?

The mechanism of Natural Language Processing involves two subsequent processes:

• Natural Language Understanding
• Natural Language Generation

Natural Language Understanding

Natural Language Understanding (NLU) attempts to comprehend the meaning of a given text. NLU requires knowledge of the nature and structure of each word contained within a text. To comprehend structure, NLU makes an attempt to resolve the following ambiguity in natural language:

• Lexical ambiguity: words have multiple meanings,
• Syntactic ambiguity: a sentence is having multiple parse trees,
• Semantic ambiguity: a sentence is having multiple meanings,
• Anaphoric ambiguity: a phrase or word which is previously mentioned but has a different meaning.

Next, the sense of each word is understood by using lexicons (vocabulary) and set of grammatical rules. However, certain different words are having similar meaning (synonyms) and words having more than one meaning (polysemy).

As we have seen with the above use cases, several attributes/artifacts are expressed in Natural Language at every stage of the software engineering lifecycle, such as commit comments, code review comments, remarks in task management and defect management tools, requirements, and test case description. Natural language attributes/artifacts have enormous potential if the information is appraised and utilized properly using NLU. For instance, natural language commit comments created by developers while checking in code might be used to denote the type of work performed by the developer.

Natural Language Generation

It is the process of automatically producing text from structured data in a readable format with meaningful phrases and sentences. Natural language generation is a difficult problem to solve. Natural language generation divided into three proposed stages:

• Text Planning: ordering of the primary content in structured data is done.
• Sentence Planning: the sentences are combined with structured data to represent the flow of information.
• Realization: finally, grammatically accurate sentences are generated to represent the material.

• Pattern Recognition : This technique compares log messages to those recorded in a pattern book in order to filter out log messages. In quality engineering, pattern recognition can assist in detecting frequently occurring system failure patterns, which can then be considered candidates for auto diagnosis and recovery via solution framing.
  
  
• Text Normalization : Normalization of text messages is the process of converting disparate messages to a common format. This is done when separate log messages use distinct terminologies but originate from the same source, such as programs or operating systems. By normalizing text, we seek to reduce its variability and bring it closer to a predefined "standard." This enables us to limit the amount of varied data that the computer must process, hence increasing efficiency. The purpose of normalization techniques such as stemming and lemmatization is to reduce a word's inflectional and occasionally derivationally related forms to a single base form. Text normalization can aid in the streamlining of analysis performed on natural language sources such as comments, descriptions, and so on, in order to conduct analysis for information retrieval.
  
  
• Automated Text Classification and Tagging : Text classifiers can be used to organize, arrange, and categorize virtually any type of text - from documents, studies, and files, to text found throughout the web. Text Classification & Tagging is the process of ordering and tagging text using a variety of keywords for subsequent analysis. In software engineering, text classification can be used to quickly and cost-effectively structure a variety of relevant text, documents, social media, chat bots, and surveys. This enables time savings when evaluating text data, automation of corporate processes, and decision-making based on data.
  
  
• Artificial Ignorance : It is a technique that makes use of Machine Learning algorithms to eliminate irrelevant log messages. It is also used to detect anomalies in system functionalities and software operations, enabling engineers to generate tailored alerts in real time in the event of a server/system malfunction or security breach.
  
  

Using NLP involves some basic NLP operations listed below:

1. Data Collection: Collect relevant data.
2. Segmentation & Tokenization: Segmentation of text into components. The process of breaking down a text paragraph into smaller chunks such as words or sentence.
3. Text Cleaning: Removal of unnecessary elements.
4. Vectorization & Feature Engineering: Transformation of the text to numerical vectors.
5. Text Lemmatization & Steaming: Lemmatization considers the context and converts the word to its meaningful base form, stemming removes the last few characters, often leading to incorrect meanings and spelling errors to reducing inflections for words.
6. Training Model: Train a model using ML algorithms to solve NLP problem.
7. Interpretation of the Result: Interpret and validate outcomes of the trained model.

Widely used NLP Libraries

There are many libraries, packages, tools available in market. Each of them has its own pros and cons. As a market trend Python is the language which has most compatible libraries. Below table will gives a summarized view of features of some of the widely used libraries. Most of them provide the basic NLP features mentioned earlier. Each NLP libraries were built with certain objectives; hence it can be said that, a single library might not provide solutions for everything.

Challenges in Natural Language Processing

In Quality Engineering, scope of applying NLP techniques to natural language data depends on multiple factors, which need to examined and evaluated before implementation. Some key challenges faced during implementation can be listed as below:

Data Challenges:

• The main challenge is information overload, which poses a big problem to access a specific, important piece of information from vast datasets. Semantic and context understanding is essential as well as challenging for summarization systems due to quality and usability issues. Also, identifying the context of interaction among entities and objects is a crucial task, especially with high dimensional, heterogeneous, complex and poor-quality data.
• Data ambiguities add challenges to contextual understanding. Semantics are important to find the relationship among entities & objects. Entities & object extraction from text and visual data could not provide accurate information unless the context and semantics of interaction are identified.
• The next challenge is extraction of relevant & correct information from unstructured/ semi-structured data using Information Extraction(IE) techniques. It is necessary to understand the competency and limitations of the existing IE techniques related to data pre-processing, data extraction and transformation, and representations for vast volumes of multidimensional unstructured data. Higher efficiency and accuracy of these IE systems are very important. But, the complexity of big and real-time data brings challenges for Machine Learning based approaches, which are dimensionality of data, scalability, distributed computing, adaptability, and usability. Effectively handling sparse, imbalance and high dimensional datasets are complex.

Text related challenges:

• Large repositories of textual data are generated from diverse sources. Though ML and NLP have emerged as the most potent and most used technology applied to the analysis of the text and text classification remains the most popular and the most used technique. Text classification could be Multi-Level (MLC) or Multi-Class (MCC). In MCC, every instance could be assigned to only one class label, whereas MLC is a classification that assigns multiple labels to a single instance.
• Solving MLC problems requires an understanding of multi-label data pre-processing for big data analysis. MLC can become very complicated due to the characteristics of real-world data such as high-dimensional label space, label dependency, and uncertainty, drifting, incomplete and imbalanced. Data reduction for large dimensional datasets and classifying multi-instance data is also a challenging task.
• Then there are the issues posed by a language translation. The main challenge with language translation is not in translating words, but in understanding the meaning of sentences to provide an accurate translation. Each text comes with different words and requires specific language skills. Choosing the right words depending on the context and the purpose of the content, is more complicated.
• A language may not have an exact match for a certain action or object that exists in another language. Idiomatic expressions explain something by way of unique examples or figures of speech. Most importantly, the meaning of particular phrases cannot be predicted by the literal definitions of the words it contains.
• The standard challenge for all new tools, is the process, storage and maintenance. Building NLP pipelines is a complex process — pre-processing, sentence splitting, tokenization, parts of speech [POS] tagging, stemming and lemmatization, and the numerical representation of words. NLP requires high-end machines to build models from large and heterogeneous data sources.