An Analysis of Word Embeddings and Prompting Techniques on PubMedQA

University of Lincoln NLP Assignment (Scored 87%)

• Natural Language Processing

• PyTorch
• Scikit-learn
• HuggingFace
• GloVe
• Word2Vec
• Llama3.2

Overview

For my Natural Language Processing module at the University of Lincoln, one of my assignments was centered around PubMedQA, a dataset comprising of medical research questions from PubMed abstracts. The task was given a question and the abstract it comes from, predict two things: a short answer (yes, or no) and a longer, detailed explanation justifying the answer.

The assignment was split into two experiments. The first experiment looks at the short-answer tasks and analyses which matters more: the type of embedding model (static or contextual), or what data the model was pre-trained on. The second looks at how much prompt engineering can affect an LLM's ability to generate an explanation.

Experiment 1: static versus contextual embeddings

I trained a logistic regression classifier on top of embeddings from four models, and a stratified random chance baseline. The four models were split across two axes: embedding type and pre-training domain.
• glove-wiki-gigaword-300, static, general-purpose, pre-trained on Wikipedia
• pubmed2018-w2v-400D, static, pre-trained on 27 million PubMed abstracts
• all-MiniLM-L6-v2, contextual, general-purpose, pre-trained on over a billion sentences
• pubmedbert-base-embeddings, contextual, pre-trained on PubMed title-abstract pairs

My hypothesis was that pubmedbert-base-embeddings would perform best, because it combines context aware embeddings with pre-training on medical domain knowledge.

Domain pre-training produced a much bigger improvement than switching from static to contextual embeddings. Moving pre-training materials from general to medical data improved weighted F1 by 12.0% for the static models and 7.3% for the contextual ones, whereas moving from static to contextual within the same domain improved weighted F1 by 4.9% for the general models and only 0.22% for the medical ones. Static embeddings gained more from domain pre-training than contextual embeddings did, which is the opposite of what I expected going in.

Experiment 2: prompting strategies for the long answer

The second experiment used an LLM (Llama-3.2-1B-Instruct fine-tuned on medical QA) to generate the long answer and compared how much prompt engineering changed the output quality. I tested five prompting techniques, each with and without repetition, giving ten prompts in total: zero-shot, one-shot, class-few-shot (one example per answer class), chain-of-thought, and role prompting. Outputs were scored with BERTScore, which compares meaning rather than exact wording, since two correct answers can be phrased completely differently.

My hypothesis was that prompts with more context would score higher, so class-few-shot ahead of one-shot ahead of zero-shot.

Class-few-shot came out on top with a BERTscore of 0.8194, but only just. Zero-shot and chain-of-thought both scored 0.8172, suggesting that prompt structure has far less influence on this task than the choice of model.

Takeaways

As made clear in both of the experiments: the material that a model is trained on mattered more than the architecture that it was built on. A static embedding model trained on the right data outperformed a contextual model trained on the wrong data.

This generalises beyond NLP. Whatever the model its ceiling is set by how well the data it learned from represents the problem you want it to solve.