More Experimentation with Embeddings
We investigate the poor performance of the composite model with external embeddings.
Changing embedding generation
First, we train the embedding for longer - 15 epochs in (1) compared to the default 5 epochs in (2). The embedding size remains at 32, and the embedding is trained on the corpus of cleaned text as before.
Changing the model hyperparameters
Second, we prevent the embedding layer from being trained further during the composite model’s training.
Finally, we train the composition model for longer: 15 epochs rather than the previous 7 epochs.
New results with FastText
The results with the new model are respectable and notably better than the older model: accuracy of 97.24% (vs. 96.27%), F1 score of 71.32% (vs. 53.70%), AUC score of 95.55% (vs. 92.74%).
Using Word2vec instead of FastText
We find that the model’s performance improves when switching from using FastText to Word2vec (3) while holding all the other hyperparameters constant. We get an accuracy of 97.99%, an F1 score of 81.12%, and an AUC score of 97.31%.
Clearly, the classification performance of the model is sensitive to the embedding technique and length of training. Unfortunately, there does not appear to be an easy way to judge the quality of an embedding on esoteric corpora (4).