After developing the sentiment analysis pipeline, it was necessary to validate its accuracy against a standard dataset. We used a dataset from Amazon reviews that had been cleaned and manually labeled with 10,000 feedbacks and had been used in a number of researches. After doing sentiment analysis and evaluating the analyzer's average accuracy, an 85 percent score was obtained. Thus, for input gathered via multiple internet sites, this was deemed to represent the accuracy. Figure depicts the user interface for the emotion analyzer.
Figure: Homescreen of Capgemini’s Sentiment Analysis Implementation
Additionally, we have completed a significant number of deployments for banking and insurance clients. One such client was a large insurance company headquartered in Australia. They presented us with 1300 user reviews of their application. We supplied them with a complete analysis in the form of a report that included the amount of positive, negative, and neutral reviews, as well as a more nuanced description of how many were extremely positive, positive, neural, negative, and extremely negative. Additionally, the research included a list of the most frequently used terms for favorable and negative comments. The report included a pie chart indicating the percentage of words that are good, negative, or neutral, as well as a similar pie chart for the reviews. Finally, we presented an idea of the general sentiment. These are depicted in the accompanying figures.
Figure: Analysis Results divided as Positive, Negative and Neutral in (a) and in fine grained manner as very positive, positive, neutral, negative and very negative in (b)
Figure: Breakdown in terms of percentage of words corresponding to positive, negative or neural sentiment and percentage of feedback which corresponds to positive, negative or neutral
Figure: Screen displaying critical words with the size of the word proportional to number of times it is used
Figure: Screen displaying popular words with the size of the word proportional to number of times it is used
Figure: Sentiment Meter displaying overall sentiment
Using this approach, we could gain insights from social media conversations and measure company and product reputation via a sentiment index. This allowed us to promptly address negative sentiment to impact the perception. Additionally, it allowed us to identify and engage top influencers to reinforce desired messages.
While sentiment analysis offers valuable insights, it's not an exact science. Sentiment tools are not precise in discerning sarcasm and wit. Due to a lack of standards, different tools can deliver different sentiment scores. One of the leaders in the sentiment analysis space, Seth Grimes, argues that this is an "80% solution."