Comparative Evaluation of GPT Models in FHIR Proficiency

Details

This academic paper, titled 'Comparative Evaluation of GPT Models in FHIR Proficiency,' assesses the capabilities of generative AI models against FHIR (Fast Healthcare Interoperability Resources), a foundational standard for healthcare data exchange. The authors evaluated GPT-3.5, GPT-4.0, and custom models like the “FHIR Interop Expert” across two FHIR examination scenarios. Novel metrics were employed, including Token Processing Cost (TPC), Accuracy-Adjusted Token Processing Cost (ATPC), Comprehensive Performance Index (CPI), and Quality-Adjusted Performance Score (QAPS). The findings indicated that GPT-4.0 exhibited superior accuracy and robustness, while custom models showed strength in domain-specific tasks through effective prompt engineering. Crucially, however, none of the tested models consistently achieved the $\geq 99\%$ accuracy required for high-stakes healthcare applications. This research provides a replicable framework for assessing AI readiness, emphasizing the necessity of refining domain-specific training and evaluation methods to ensure the responsible and effective integration of AI into clinical workflows.