Understanding what users think, or at the very least write, about our apps can assist us in identifying areas for improvement and providing more testing ideas. While sentiment analysis is a minor subfield of quality engineering, it can provide useful information to complement our test strategy. This chapter comes with a description of the procedure we used to conduct sentiment analysis.
Arguably, consumer reviews were invented in 1999 by the website rateitall.com. Since that time, reviews have evolved into a widely acknowledged method of discovering what others think and how they rate items and services. They have an impact on practically every part of our lives, whether it is purchasing a gift, planning a vacation, or selecting a restaurant. Consumers can respond with their preferred emoticon, gripe about or praise a product on their social media profile or on an online review portal such as Yelp or Google reviews, or start discussions on numerous forums.
The internet has empowered customers to have their voices heard. While in an ideal world, businesses would truly listen to their consumers and include their feedback when improving their product or service offerings, this is not frequently the case. Consider the situation of a newly released application: millions of people will download it, and at least 10% of those who download it will leave a review. Thus, with so many people offering their opinions across multiple internet platforms, it becomes a tremendous endeavor for firms to physically comb through each one and record the feelings.
Reviews - more than ratings - complement development and testing by bridging the gap between your expectations and the perceptions of a system or application in production. Reviews can be a valuable source of continuing information for the entire team, including those responsible for product design, development, testing, operations, monitoring, and customer support. Customer support and engagement may place a greater emphasis on users' feelings and sentiments, ensuring they receive a kind and attentive ear. And we appreciate their feedback on how we might enhance our apps and service.
Here is when sentiment analysis comes in handy. It can be used as a proxy for measuring user’s perceptions. Sentiment analysis is the method by which we attempt to comprehend a person's affective state of mind, which is done through the use of Natural Language Processing and text processing. Sentiment analysis is a critical technique for determining how customers feel about a product and for determining an effective strategy to improve the product's quality. When firms introduce a new product, they attempt to gauge customer sentiment by collecting written comments from buyers within a few days of the product's purchase. As business consultant David J. Greer puts it, "a consumer speaking positively about your company is worth 10 times what you write or say about yourself." The same is true for any product, and these reviews will help to increase the product's trustworthiness.
For instance, mobile app stores are an important source of feedback because they are a well-known gathering place, influential, and reasonably easy to mine for usable data. App store comments can include both worthwhile recommendations and requests from consumers, as well as reports of issues that may go overlooked by the development team.
Sentiment analytics examine a user's communication with the app in order to ascertain the sentiments the user wishes to convey to others. Satisfied and happy users may advocate and promote an app, but dissatisfied and unhappy customers may attempt to dissuade others from utilizing an app, business, or service. Businesses may be required to respond swiftly, appropriately, and positively to both positive and negative sentiments. There are numerous difficulties associated with working with textual data, specifically with regard to online feedback regarding a particular app. These include the volume of feedback, which can reach several thousand items each day for major Google and Facebook apps, spam and unethical feedback (including paid-for evaluations that artificially inflate the overall rating), dealing with numerous languages, and deciphering what people write. Certain types of feedback may be misconstrued. Sentiment analysis, for example, could have difficulty correctly interpreting negativity, sarcasm, and slang.
Sentiment analysis of user feedback can be used to assist in identifying and rectifying gaps in product documentation – as well as in the training and materials utilized by the customer service team. This ensures that assets (help documentation and trained support agents) are in place to address the most common user issues and adapted to shifting patterns as an application matures.
There are three methods for performing sentiment analysis.
The processes that was followed is illustrated in the figure below:
Figure: Sentiment Analysis
The sentiment analysis pipeline begins with the collection of feedback from various channels. This is accomplished through the use of a Spider, which crawls through the appropriate platforms and receives feedback from all of them. The collected Text Comments are then Pre-processed. During pre-processing, the text is first cleaned up and converted into a format that is suitable to be given as the input to the sentiment analyzer.
For Sentiment Analysis, we have used Valence Aware Dictionary for Sentiment Reasoning (VADER). VADER is sensitive to both polarity (positive/negative) and the intensity (strength) of the sentiment in the sentence and is based on sentiment-related word lexicons. In this methodology, each word in the lexicon is evaluated to determine whether it is positive or negative, and to what extent it is positive or negative.
When VADER examines a piece of text, it verifies whether any of the words in the content are available in the lexicon and then generates four sentiment metrics based on these word classifications. The first three, positive, neutral, and negative, indicate the amount to which the content fits within those categories. It is possible that the model sentence was classified as 44% positive, 54% neutral, and 0% negative. The final metric, the compound score, is the sum of the lexical grades normalized to fall between – 1 and 1. This value indicates if the sentence as a whole is positive, neutral, or negative. Additionally, VADER's vocabulary includes emoticons and punctuation, which enable it to adjust its ratings based on how a word is used in conjunction with other words.
After developing the sentiment analysis pipeline, it was necessary to validate its accuracy against a standard dataset. We used a dataset from Amazon reviews that had been cleaned and manually labeled with 10,000 feedbacks and had been used in a number of researches. After doing sentiment analysis and evaluating the analyzer's average accuracy, an 85 percent score was obtained. Thus, for input gathered via multiple internet sites, this was deemed to represent the accuracy. Figure depicts the user interface for the emotion analyzer.
Figure: Homescreen of Capgemini’s Sentiment Analysis Implementation
Additionally, we have completed a significant number of deployments for banking and insurance clients. One such client was a large insurance company headquartered in Australia. They presented us with 1300 user reviews of their application. We supplied them with a complete analysis in the form of a report that included the amount of positive, negative, and neutral reviews, as well as a more nuanced description of how many were extremely positive, positive, neural, negative, and extremely negative. Additionally, the research included a list of the most frequently used terms for favorable and negative comments. The report included a pie chart indicating the percentage of words that are good, negative, or neutral, as well as a similar pie chart for the reviews. Finally, we presented an idea of the general sentiment. These are depicted in the accompanying figures.
Figure: Analysis Results divided as Positive, Negative and Neutral in (a) and in fine grained manner as very positive, positive, neutral, negative and very negative in (b)
Figure: Breakdown in terms of percentage of words corresponding to positive, negative or neural sentiment and percentage of feedback which corresponds to positive, negative or neutral
Figure: Screen displaying critical words with the size of the word proportional to number of times it is used
Figure: Screen displaying popular words with the size of the word proportional to number of times it is used
Figure: Sentiment Meter displaying overall sentiment
Using this approach, we could gain insights from social media conversations and measure company and product reputation via a sentiment index. This allowed us to promptly address negative sentiment to impact the perception. Additionally, it allowed us to identify and engage top influencers to reinforce desired messages.
While sentiment analysis offers valuable insights, it's not an exact science. Sentiment tools are not precise in discerning sarcasm and wit. Due to a lack of standards, different tools can deliver different sentiment scores. One of the leaders in the sentiment analysis space, Seth Grimes, argues that this is an "80% solution."
Dr. Raghav Menon
Raghav Menon is passionate about AI/ML, Natural Language Processing, Speech processing, Image and Signal Processing. He has a PhD in Signal Processing and has several years of Research and Development experience. At Capgemini, he had been working with the Cognitive Document processing team. Currently he is attached to the Analytics COE team in Capgemini where he looks into the application of AI/ML algorithms for software testing among the other areas of application of AI/ML. He has several publications in the areas of AI/ML, Speech and Signal Processing which can be accessed at scholar.google.co.in. His last assignment had been with the United Nations, Global Pulse Labs in Stellenbosch, South Africa.
Antoine Aymer (chief editor)
Antoine Aymer is a passionate technologist with a structured passion for innovation. He is currently the Chief Technology Officer for Sogeti's quality engineering business. Antoine is accountable for bringing solutions and services to the global market, which includes analyzing market trends, evaluating innovation, defining the scope of services and tools, and advising customers and delivery teams. Apart from numerous industry reports, such as the Continuous Testing Reports, the 2020 state of Performance Engineering, Antoine co-authored the "Mobile Analytics Playbook," which aims to assist practitioners in improving the quality, velocity, and efficiency of their mobile applications through the integration of analytics and testing.
Part of the Capgemini Group, Sogeti operates in more than 100 locations globally. Working closely with clients and partners to take full advantage of the opportunities of technology, Sogeti combines agility and speed of implementation to tailor innovative future-focused solutions in Digital Assurance and Testing, Cloud and Cybersecurity, all fueled by AI and automation. With its hands-on ‘value in the making’ approach and passion for technology, Sogeti helps organizations implement their digital journeys at speed.
Visit us at www.sogeti.com
Capgemini is a global leader in partnering with companies to transform and manage their business by harnessing the power of technology. The Group is guided everyday by its purpose of unleashing human energy through technology for an inclusive and sustainable future. It is a responsible and diverse organization of 325,000 team members in nearly 50 countries. With its strong 55 year heritage and deep industry expertise, Capgemini is trusted by its clients to address the entire breadth of their business needs, from strategy and design to operations, fueled by the fast evolving and innovative world of cloud, data, AI, connectivity, software, digital engineering and platforms. The Group reported in 2021 global revenues of €18 billion.
Get the Future You Want | www.capgemini.com