Sentiment Analysis using VADER
Sentiment analysis is one of the common applications of Natural Language Processing (NLP) techniques used for understanding people’s opinions, sentiments or emotions expressed through text. Most often, it is used by organizations to get insights on customers’ perception on products/ services offered, and thereby take suitable actions to retain/improve the sentiments. It also helps organizations to assess employee engagement levels through feedbacks collected via open-ended surveys or understand people’s sentiments expressed on various topics through social media platforms.
Social media texts, which are generally informal, are different from the reviews on products/services. Opinions on social media are expressed using acronyms and slangs which may not be present in product reviews. While deep learning techniques have improved the quality of analyzing sentiments from these texts, they require large dataset with annotations for training. Gathering relevant data and annotating them without bias, costs time and effort.
So, if you are working on informal text analysis, and do not have required data or the time to annotate them, various open source algorithms like NLTK’s TweetTokenizer, TextBlob, Affin, PySentiment , Spacy ,VADER etc help to obtain the sentiments of input text directly without building sophisticated classification models.
I have used VADER — Valence Aware Dictionary for sEntiment Reasoning, for analyzing informal texts when I didn’t have large dataset of annotated sentences, and realized that it works quite well, in comparison to other aforementioned algorithms.
VADER is a rule based algorithm, that makes use of a lexicon and rules that consider semantic orientation of words, degree modifiers, word-shape amplifiers, punctuation amplifiers etc to identify sentiments and quantify their intensity.
VADER lexicon contains acronyms, slangs and emojis used in social media platforms, in addition to formal English words. Each token in the lexicon is associated with raw sentiment ratings, its mean and standard deviation. The raw ratings were assigned on a scale from -4 to +4, by ten independent humans who were pre-trained and assessed for inter-rater reliability.
The algorithm provides scores based on positive, negative and neutral proportion of sentiments expressed in text. Compound score, which is the aggregate score representing the sentiment of the entire text, is also provided as the output. It is computed by “summing the valence scores of each token in the text, adjusted based on rules and normalized to be between -1 and +1 where -1 indicates extreme negativity and +1 indicates extreme positivity”. While the algorithm itself performs well, the users can enhance the lexicon by adding tokens or modifying the rules based on custom needs. They can also set thresholds on compound scores for classifying sentences as positive, negative or neutral or scale it as required.
Let’s see a few examples :
#importing library
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
# Analysing text
text = ["It was a great movie" ,
"Make sure you :) or :D today!" ,
"The plot sux"]
analyzer = SentimentIntensityAnalyzer()
for txt in text:
score = analyzer.polarity_scores(txt)
print("{:.<50} {}".format(txt, str(score)))
We can observe that the output gives us the proportion of polarities along with compund score, as mentioned earlier. The compund score clearly denotes if the overall sentiment is negative, positive or neutral.Note that the slang and emojis are handled correctly.
Scaling Scores:
Now, let’s write a function to get the score on a scale of 10 (0 being extremely negative, 5 being neutral and 10 being extremely positive).
def scale_score(text) :
for txt in text:
ss = analyzer.polarity_scores(txt) # get the dictionary of scores
score=round((ss['compound']+1)*5,2) # extract the compound score and scale it
print("{:.<50} {}".format(txt, str(score)))On executing this function, we get the below result :
We see that the results are scaled as desired.
Upating the lexicon:
Now, if we wish to update the lexicon, it can be done easily. Let’s look at an example before and after adding a token to the lexicon
text = ["Weather is worse",
"Our house was demolished in the earthquake"]
I wish to make the first sentence more negative and make the second negative (5 indicates neutrality). We can do this by calling the update function after initializing the analyzer object in our code. Here, I am adding a new token , ‘demolished’ and adjusting the mean rating for the existing token, ‘worse’.
analyzer = SentimentIntensityAnalyzer()
analyzer.lexicon.update({'demolished' : -2.5 , 'worse' : -3})Now, let’s execute the scoring function after executing the above code snippet and observe the results.
Thus we see that our objectives of making the sentences more negative are accomplished. This can also be done by directly updating the lexicon file, but , to maintain consistency, it is advised to follow the steps suggested by the authors.
Using thresholds :
Sometimes, we may be interested in classifiying the sentences as Extreme Dissatisfaction, Neutral etc. This can be done by using thresholds on compound score. Example : Sentences with scores less than2.5 can be considered as expressing Extreme dissatisfaction, between 2.5and 4.5 as Dissatisfaction, 4.5 and 5.5 as neutral and so on. The thresholds can be set on the scores obtained without scaling as well.
Apart from these, the rules in the algorithm can also be added/ modified to get better results. Thus it gives the developers the flexibility to finetune the results based on the needs.
References:
- Hutto, C.J. & Gilbert, E.E. (2014). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media (ICWSM-14). Ann Arbor, MI, June 2014.
- https://www.geeksforgeeks.org/python-sentiment-analysis-using-vader/#:~:text=VADER%20uses%20a%20combination%20of,or%20negative%20a%20sentiment%20is.
