DavidGadyan

Data Science & ML

LDA & LIWC Analysis of Tweets

By David Gadyan · Published · Updated · Client: Hamad S.

pyLDAvis screen: intertopic distance map of four topics and top terms bar chart led by covid

Text-analytics study for Hamad S. on Trump's tweets about Covid and a few other topics. After exploratory analysis and preprocessing in Python, I ran LDA topic modelling and delivered the discovered topic clusters as interactive pyLDAvis HTML files, then scored the corpus with the LIWC2015 dictionary, after resolving a dictionary encoding problem with its author, Dr. James W. Pennebaker.

The problem

The client wanted to understand what Trump was tweeting about during Covid, and on a few other topics, in two complementary ways: which themes the tweets clustered into, and what the language itself revealed psychologically. That called for two different methods on the same corpus, LDA topic modelling for the themes and LIWC for the psycholinguistic scoring.

What I built

  • Exploratory data analysis of the tweet corpus, followed by text preprocessing to get it into shape for modelling.
  • LDA topic modelling in Python, with the discovered clusters exported as interactive pyLDAvis HTML files so the client could explore each topic, its top terms and how topics relate to one another in a browser.
  • LIWC analysis in Python using the LIWC2015 dictionary, producing the linguistic and psychological category scores for the corpus.
  • A fix for the dictionary itself: the LIWC2015 file we had was encoded in a way the tooling could not read, so I contacted the author of the work, Dr. James W. Pennebaker, who kindly sent a decoded version.

Outcome

The client received interactive topic-cluster visualisations alongside LIWC scoring for the full corpus, which together give both a thematic and a psycholinguistic read of the tweets. Many thanks to Dr. Pennebaker for unblocking the LIWC part.

Related projects

← All projects