Work / research
Benchmarking Retrieval for Medical RAG
Head-to-head evaluation of sparse, dense and graph-based retrieval on a PubMed medical corpus, run at scale on WPI's Turing cluster.
- Kind
- research
- Status
- ongoing
- Role
- undergraduate researcher
- Period
- Jan 2026 – present
- Affiliation
- WPI, Prof. Gerych's group
Problem
A retrieval-augmented generation system is only as good as the documents it retrieves, but “which retriever?” is usually answered by convention rather than by measurement. In the medical domain a bad retrieval is not a bad answer, it is a wrong one. This work asks whether the retrieval methods people default to actually hold up on medical corpora, where terminology is dense, synonymy is rampant and the cost of a miss is high.
Data
- Corpus: PubMed-derived medical document corpus (Hugging Face)
- Evaluation: SCT-Bench
- Splits:
- Known biases:
Approach
The comparison holds the generator fixed and varies only the retriever, so any difference in downstream answer quality is attributable to retrieval:
- TF-IDF: sparse lexical baseline
- BM25: the standard sparse benchmark
- Graph-based retrieval: exploits structure between documents rather than treating the corpus as a bag of independent passages
Benchmarking jobs are dispatched as Slurm batch arrays on WPI’s Turing ARC cluster, which is what makes a full sweep across retrievers × query sets tractable rather than a weekend of laptop time.