Exploring Lakehouse//RT and Reyden: Can Databricks Handle FHIR Data at Scale?
DatabricksとReydenでFHIRデータを大規模に処理：課題と性能検証の結果
Summary
A healthcare client sought to manage FHIR data within their existing data lakehouse (Databricks). The article proposes an architecture using the Lakehouse//RT engine, demonstrating its high processing capability without needing a dedicated server.
Details
This technical deep dive addresses challenges and solutions for handling FHIR data in the healthcare sector. The client wanted to keep operations within Databricks but faced issues with setting up dedicated FHIR servers (like HAPI), which would introduce data duplication, new governance boundaries, and increased operational overhead. The proposed solution utilizes a three-layer architecture: Bronze (raw ingestion), Silver (cleaned/validated data), and an API layer built on FastAPI. In this setup, the FastAPI receives FHIR client requests, generates optimized SQL, and Databricks retrieves data from Bronze or Silver before mapping results back to the FHIR schema. Initial performance tests showed that traditional Databricks SQL queries for search could take hundreds of milliseconds. However, implementing the Lakehouse//RT engine (Reyden) achieved significant improvements. Specifically, individual EOB lookups saw latency drop dramatically from an average of 550ms down to 86ms, achieving a highly usable performance level.
Original content copyright by respective publishers