Minsol Kim, Du Liu, Ragasudha Botta, Seok Gyu Han, Man Jiang, Jason Gusdorf, Pattie Maes, and Paul Pu Liang. "Exploring RAG-driven Multimodal LLMs for Explainable ECG Interpretation." In 39th Conference on Neural Information Processing Systems Workshop: Foundation Models for the Brain and Body, NeurIPS, 2025.

Abstract

Foundation models offer a path toward safer, more generalizable biosignal analysis, yet their role in clinical interpretation of Electrocardiograms (ECGs) remains unclear. We present a multimodal framework with Retrieval-Augmented Generation (RAG) that couples 12-lead ECG waveforms with structured patient metadata (age, sex) to guide large language models (LLMs) and a downstream ECG encoder. Using PTB-XL for benchmarking and MIMIC-IV-ECG for pretraining, we compare four LLMs within the same RAG pipeline—GPT-4, GPT-3.5, Claude 3.5 Sonnet, and Llama 3.2—for arrhythmia classification across five diagnostic superclasses. GPT-4 achieves the best overall performance (AUC = 0.940) and maintains stable accuracy when demographic features are ablated or added, suggesting reduced sensitivity to demographic shifts. Metadata provided the largest relative gains for models with lower performance in our benchmark, while qualitative analysis highlighted remaining failure modes such as hallucinations and retrieval mismatches. These results indicate that RAG-driven multimodal prompting can enhance ECG interpretation and fairness when paired with high-performing foundation models, and we outline practical levers—retrieval design, prompt structure, and metadata integration—for building equitable and scalable clinical ECG systems.