Dr Alexandra Huang-Kokina

The aim of this project is to compare the themes in the opera librettos of the works by two famous composers, Giuseppe Verdi and Giacomo Puccini. The primary tool for this analysis is Latent Dirichlet Allocation (LDA), a type of probabilistic model used for discovering topics from a collection of documents. In this case, the documents are the librettos of the operas composed by Verdi and Puccini.

Steps Undertaken So Far:

1. Data Collection: The first step for me was to collect the librettos of the operas composed by Verdi and Puccini from online databases. These were collected in text format and stored locally.
2. Preprocessing: I then pre-processed the librettos to prepare them for analysis using Python’s Natural Language Toolkit (NLTK) and Gensim library. This involved tokenizing the text (i.e., breaking down a given piece of text into smaller units called ‘tokens’, in my case – individual words, while in some other cases it could be characters, phrases, etc., depending on the level of granuality needed for analysis); removing stop words (common words like ‘the’, ‘and’, ‘a’, etc. that don’t carry much meaning); and converting all the text to lower case using Python’s built-in string methods.
3. Dictionary Creation: After pre-processing, I used the Gensim library to create a dictionary and a corpus, which are necessary inputs for the LDA model. A dictionary in the context of Gensim is the mapping between words and their integer ids (i.e., numerical representations of words/tokens in a text corpus). It allows the model to convert human-readable words into a format that can be processed more efficiently. I created the dictionary using the corpora.Dictionary function from Gensim, which takes a list of tokens and assigns a unique integer id to each unique word in the documents while also recording the counts of each unique word.
4. Corpus Creation: I also created a corpus which is a representation of the original documents in a format that can be used by the model. In my case, I used the Bag-of-Words model, which represents each document as a list of tuples. Each tuple contains a word id and the frequency of that word in the document. The corpus was created using the doc2bow method of the dictionary, which converts a document into the Bag-of-Words format. It was applied to each document in my list of documents.
5. Model Training: An LDA model was then trained on the corpus. I trained the model with different numbers of topics and passes to find the optimal parameters that give the highest coherence score. The coherence score is a measure of the quality of the topics generated by the model, with higher scores indicating better quality.
6. Parameter Tuning: The alpha and beta parameters of the LDA model were also tuned to improve the coherence score. Alpha represents document-topic density and Beta represents topic-word density. The higher the value of alpha, documents are composed of more topics; and the higher the beta, topics are composed of more words.
7. Topic Examination: The topics generated by the model were then examined by myself. Each topic is a combination of words, and the weight of each word in the topic indicates how important that word is for the topic.

Next Steps:

1. Improve Pre-processing: The pre-processing steps could be revisited to see if improvements can be made. This could involve more sophisticated techniques for removing stop words, lemmatization (reducing words to their base or root form), and the inclusion of bigrams or trigrams (combinations of two or three words that often appear together).
2. Experiment with Model Parameters: Further experimentation could be done with the parameters of the LDA model to try to improve the coherence score and the interpretability of the topics.
3. Try Different Models: Other topic modeling algorithms, such as Non-negative Matrix Factorization (NMF), could be tried to see if they produce more interpretable topics.
4. Topic Interpretation: Once satisfactory topics have been generated, the next step would be to interpret these topics and see what themes they represent in the operas of Verdi and Puccini. This could involve mapping the topics back to the original librettos and seeing which topics are most prominent in each opera.
5. Comparison of Themes: The final step would be to compare the themes found in the operas of Verdi and Puccini. This could involve a qualitative comparison of the themes, as well as a quantitative comparison of how often each theme appears in the operas of the two composers.