Question Answering on Patient Medical Records with Private Fine-Tuned LLMs
Question Answering on Patient Medical Records with Private Fine-Tuned LLMs - ChatPaper
Summary
Stanford University and Genloop Labs proposed a novel semantic QA approach using privately fine-tuned Large Language Models (LLMs) to extract information from Electronic Health Records (EHRs). This method first identifies relevant FHIR resources and then answers questions based on those resources, demonstrating high performance.
Details
This study addresses the challenge of extracting crucial information from complex Electronic Health Records (EHRs). These EHRs are typically stored according to the Fast Healthcare Interoperability Resources (FHIR) standard, but their sheer volume and complexity make it difficult for users to retrieve or interpret critical health information. Large Language Models (LLMs) powered by semantic QA offer a promising solution. However, ensuring confidentiality and compliance when handling medical data is paramount, necessitating private and edge deployments of LLMs. The paper proposes a two-stage approach: first, identifying the most relevant FHIR resources for a user query (Task 1), and second, answering the query based on these identified resources (Task 2). The team evaluated their self-tuned LLMs against major benchmarks like GPT-4 and GPT-4o. Results showed that despite being significantly smaller in size, the proposed LLMs outperformed the GPT-4 family models by 0.55% in F1 score for Task 1 and by 42% in Meteor score for Task 2. Furthermore, advanced aspects of LLM usage, including sequential tuning and data size impact, were examined. The models and datasets are publicly available for community validation.
Original content copyright by respective publishers