Sentiment Analysis: A Deep Dive Into the Theory, Methods, and Applications

There are various models developed to perform sentiment analysis on datasets. It is important to understand how they came to be and how they function, to ensure that the model you choose is most suited to the data you have at hand.

In this article, I will summarize the theory behind sentiment analysis, explain the rationale behind it, and show some applications of successful sentiment classification. This article is compiled as a result of a systematic literature review on topic modeling and sentiment analysis I did.

What is sentiment analysis in NLP?

Sentiment analysis as a subset of Natural Language Processing (NLP).

NLP is a field of research that studies the ability to decode data from natural language using computational means. NLP also examines how this decoded data can be incorporated into machine learning and statistical programming software.

Through NLP, computer programs can perform data analysis using machine learning algorithms on textual data, which is abundant in a public-source format. In addition, they can extract meaning from data, or otherwise — semantics.

The main aim of the NLP field is to bridge the gaps in communication between computer programs and humans. Programs are constantly improved to decode language and speech data into meaningful semantic insights through processing, analysis, and synthesis.

How can sentiment analysis be defined?

Sentiment analysis is a discipline that aims to extract qualitative characteristics from user’s text data, such as sentiment, opinions, thoughts, and behavioral intent using natural language processing methods.

As explained in Zhang et al.’s (2018) study on sentiment analysis, social media texts are particularly useful for sentiment analysis research as they are:

used to express a standpoint
filled with subjective text.

Social media texts are defined in academic literature as short-form texts. This type of text is more challenging to do sentiment analysis on, as there is less context for the model to work with. In comparison, sentiment analysis performed on long-form text, such as news articles, is less challenging.

The effectiveness of the sentiment extraction in short-form text relies on the application of more advanced methodologies, such as deep convolutional neural networks.

Social media data in particular requires more complex methods in information retrieval due to the use of creative language, slang, and abbreviations. The presence of these can impact the efficiency of the sentiment analysis model, if not accounted for.

How is sentiment analysis done?

Traditional studies on sentiment analysis aim to detect polarity in a given text by classifying it as positive, negative, or neutral.

This categorization need is considered one of the key limitations to traditional sentiment analysis. Some academics argue that sentiment analysis in its traditional form is unable to address the complexities of modern-day expression, as it fails to capture objectivity and subjectivity. For instance, if a model is tasked to categorize between fake news or someone’s opinions and facts, traditional sentiment analysis will not be sufficient.

More advanced methods attempt to recognize multiple differentiated affective manifestations in text, which indicate emotions and opinions through analysis of the language used for self-expression.

Such methods often aim to simultaneously detect and extract topic models. For this reason, deep learning approaches such as convolutional neural networks (CNNs) are often used. CNNs are also used in sentiment analysis of short-form texts, as indicated in multiple research papers (e.g. Dos Santos and Gatti (2014), Kale et al. (2018), and Tang et al. (2015)).

What models can be used for sentiment analysis?

Sentiment analysis can be done via supervised, semi-supervised, and unsupervised machine learning algorithms.

Supervised machine learning models are the most difficult to obtain data on for sentiment analysis, as it requires labels for a subset of the data, with which to train the model.

Semi-supervised approaches utilize a small number of labeled samples as training data as means of improving classification accuracy, with an example being the model published by da Silva et al. (2016), where Twitter data is classified using a Support Vector Machines (SVM) classifier.

What text can sentiment analysis be performed on?

Sentiment analysis can be performed at a document level, sentence level, and aspect (word) level.

Short-form texts, such as content from social media are best analyzed with sentiment analysis at a sentence level as they usually consist of a single or few sentences.

Some models analyze individual words with the assumption that words in the same sentence share the same emotion. Such an approach is Tang et al.’s (2019) hidden Topic-Emotion Transition model, which models topics and emotions in successive sentences as a Markov chain.

This approach enables the simultaneous detection of document-level and sentence-level emotion.

Can sentiment analysis be performed on another type of data, besides text?

Multimodal sentiment analysis has grown as a field in recent years, with models proposed in the area taking advantage of recent developments in weakly supervised deep learning approaches.

Multimodal event topic modeling has also emerged, which has been demonstrated as promising for the area of predictive analysis of consumer behavior and sociology.

Together, topic modeling and sentiment analysis in a multimodal context are recognized as a way of improving human-agent interactions. An example of how this is applied is the area of automatic speech recognition.

What are the two main ways to do sentiment analysis?

#1 Using a pre-developed, manually-built sentiment lexicon.

Sentiment analysis has initially been performed using pre-developed, manually built sentiment lexicons.

Such lexicons are:

Subjectivity Wordlist
WordNet-Affect
SentiWordNet
SenticNet
AFINN
Sentiwords
SO-CAL
Opinion lexicon
WordStat

Each of these has a various scale of rating and various word counts.

Such lexicons have been used as foundations for model development, with examples being

Polarity Classification Algorithm (PCA), which classifies tweet sentiment
Enhanced Emoticon Classifier (EEC)
Improved Polarity Classifier (IPC)
SentiWordNet Classifier (SWNC).

Amongst these, superior performance is demonstrated by the PCA.

Benefits and limitations of this approach

These approaches are useful in:

distinguishing subjective or objective speech
categorizing sentiment as positive, negative, or neutral.

As a limitation, they only enable researchers to extract sentiment primarily from the perspective of the writer as opposed to the reader.

#2 Using a Machine Learning Approach

Except for lexicon-based approaches, sentiment analysis can be performed using a machine learning approach.

This can be done via statistical models trained on human-annotated datasets, hence — utilizing semi-supervised learning.

Another way is to combine multiple shallow machine-learning approaches. For instance, for sentiment analysis of tweets, Ahuja et al. (2019) conclude that TF-IDF performs better as compared to N-Grams in terms of feature extraction. The combination of TF-IDF with logistic regression is considered most efficient among the studies sample of their paper.

Ensemble classifiers are also shown to be a good way to solve one of the limitations of lexicon approaches. Specifically, a 2018 study approaches the problem of multi-label sentiment classification from the perspective of the reader, applying a model to a news dataset. The study demonstrates the superiority of ensemble classifiers when compared to other methods.

As mentioned previously, deep learning techniques can also be applied in sentiment analysis.

Benefits and Limitations of this approach

Each perspective offers its limitations and opts for compromising either the accuracy or generalisability of the analysis. Here is a summary of the benefits and limitations of common sentiment analysis machine learning approaches:

N-gram models (or conditional random fields) work via sequence tagging (e.g. part of speech tagging or shallow parsing). They capture work order and grammar well. Their disadvantage is high feature dimensionality.
Semi-supervised learning approaches are good for quick and easy determination of the polarity of text. They are not so good at determining subjectivity or objectivity in the text.
Deep Learning approaches are good at meta-level feature extraction with large datasets and typically perform better than N-gram models. They do not perform well when there is a lot of noise in the data, such as slang, abbreviations, or even emojis. This makes them ill-suited to social media sentiment analysis.
Multiple kernel learning models work when features are organized into groups. This enables multimodal sentiment analysis, but the computation is slow.

The main challenge in sentiment analysis is language ambiguity or the fact that many times when language is spoken, there might be a presence of mixed semantic attributes. This makes it difficult for a classification algorithm to perform its function.

It is also difficult for people who will view the program’s output and classification to categorize the context in which the semantics are given, which might hinder the effectiveness of the classification overall and its usability in a real-world context.

Even so, it is considered that these hindrances are outweighed by the benefits that sentiment classification offers in terms of speed in comparison with human evaluation and insight.

Why should companies implement sentiment analysis?

There can be many reasons why companies would like to tap into sentiment analysis and natural language processing technologies.

Based on my research, I’ve summarised five main reasons, for which I will also be providing examples of how this is done in real life.

#1 Increase competitive advantage

The implementation of sentiment analysis and predictive behavior modeling techniques is considered a source of competitive advantage for organizations and is recommended by scholars.

#2 Evaluate the power of a company’s consumer network

Sentiment Analysis can also be used in measuring the power of the consumer’s network. This relates to measuring how efficient word-of-mouth marketing can be.

It can also be used to track individual recommendations given amongst members of online societal groups. This can enable companies to target consumers with personalized web-ads, based on the recommendation given by their peers.

#3 Utilize public, user-generated, and readily available data

Textual data is also, more available than numeric and it can be argued that high-level human language contains a huge amount of complexity and nuance.

This has led to sentiment analysis being researched as a potential solution to a variety of business problems in various contexts such as pattern recognition of social media for the detection of road accidents or stock market prediction, to name a few.

#4 Identify patterns and make accurate, data-driven predictions about market changes

Asur and Huberman argue deep understanding of social media communications can aid accurate predictions of the outcome of future events. The example they provide is looking at real-time analytical software using Twitter feeds data for predicting box-office success. This outperformed Hollywood Stock Exchange information.

Twitter data has also been used for cluster analysis by a cognitive pattern recognition system, which picked up real-time information on happening road-traffic events prior to any mainstream reporting channels.

Similar studies help affirm that social media platforms are a pool of collective wisdom. Thus, companies are incentivized to apply machine learning for data mining and behavioral prediction to accurately inform future actions of business users for various purposes:

market prediction — e.g. stock market
outcome forecasting — e.g. for political elections
crisis management
understanding individual and collective emotional response communication — a similar approach was implemented in one of Lidl’s social media campaigns

All of these studies anticipate errors in forecast upon conducting intelligent analytical processes on human emotion, response, and social behavior, due to the unpredictable nature of us, humans, and have noted this as a model limitation.

#5 Efficiency, processing speed, and reduced human errors

Due to the existing constraints of machine learning software in performing text analytics, companies currently have intense human-resource-related spending for staff to go manually validate data. Yet there are even further inefficiencies when humans do these tasks, themselves, mostly related to the time needed to shift through data.

Just imagine how inefficient would it be for anyone to go through thousands of tweets and classify them!

This presents an opportunity to create value by solving existing business problems through continuous improvement of the models, operating in the industry. The implementation of an autonomous solution could reduce the risk of human error in the interpretation of the data.

How to get started with sentiment analysis in Google Sheets ✨

A common misconception is that you need coding skills to start doing sentiment analysis. This couldn’t be further from the truth. You can get started analysing sentiment in text in Google Sheets, using the Google Natural Language API. Use the Google Sheets template linked in my resource section for Sentiment Analysis, for which you will only need an API key and some text to analyze.

To get started, follow these simple steps:

get your Google Natural Language API key
copy the Google Sheets template
Click on the Extensions menu from the top then AppScript, then enter your API key in line 4 of the code. Then click SAVE project
Return to the sheet. Paste the text data in column A
Find the menu in the top-nav that says Sentiment Tools > Mark entities and Sentiment and click it.
Your data will populate in the pink columns and is ready for your to visualize (check out my Looker Studio dashboard if you need help with the visualization of the data).

This type of analysis can be very useful, especially in digital marketing and consumer research as it can be used for things like online reputation management, brand sentiment monitoring, or social comments sentiment analysis. To see some practical ways to incorporate sentiment analysis in your digital marketing or SEO strategy, check out this blog post: 7 Practical ways to implement sentiment analysis in digital marketing and SEO.

Final Thoughts

One of the challenges, faced by natural language processing and machine learning, in general, is that useful and problem-relevant information is often seeded in a large pool of chaotically clustered data. Another challenge is that the data might not provide any insight or might be considered useless in relation to the business problem when approached for consideration by a human agent.

These challenges are also reflected in sentiment analysis, with it being a subset of NLP.

This has sparked a wave of research into understanding the constructs of language and mining from it: user intent, sentiment, and subjectivity.

Such models are targets for organizations due to the potential of:

increased competitive advantage,
ability to predict behavior and response,
ability to better target consumers at different stages of their consumer journey
the potential of reduced need for human involvement, thus greater efficiency, and reduced operational costs.

Through monitoring of public sentiment, companies can become more adaptive to the market.