Configurable Clinical Information Extraction with Agentic RAG: What Works, What Breaks, and Why
Abstract
ACIE, an agentic RAG system deployed in a clinical setting, demonstrates high accuracy in extracting medical information from complex patient contexts, achieving 96.5% acceptance rate by nuclear-medicine physicians across 7,326 judgments.
Patient contexts span hundreds of heterogeneous documents and thousands of structured data points, yet the document-level metadata that AI systems need for retrieval and triage is absent or incomplete. Standard retrieval-augmented generation fails on this data, mishandling temporal reasoning, cross-document dependencies, and missing metadata. We deploy ACIE (Agentic Clinical Information Extraction) at University Medicine Essen: an on-premise agentic RAG pipeline that reasons over complete patient contexts and grounds every answer in source passages for clinician verification. We quantify the metadata gap, trace the architectural decisions it shaped, and evaluate extraction alongside an independent retrospective lymphoma registry study, in which nuclear-medicine physicians verify every extracted value against its cited sources. Across 7,326 judgments, clinicians accepted 96.5\% of extractions, with per-type acceptance ranging from 80\% to 99\%.
Community
We deploy ACIE (Agentic Clinical Information Extraction) at University Medicine Essen, an on-premise agentic RAG system that extracts structured data from complete patient contexts spanning hundreds of documents and thousands of FHIR resources. Clinicians configure what to extract through typed schemas, with no developer involvement.
Key findings from our deployment on one of the largest clinical FHIR repositories in Europe (~2B resources, 1.7M patients):
- Across 7,326 clinician judgments in a retrospective lymphoma registry study, physicians accepted 96.5% of extractions, with per-type acceptance ranging from 80% to 99%
- We quantify the metadata gap in real clinical data: 56.5% of documents carry timestamps outside their encounter period, standard document-type codes cover only 2.4% of documents, and a third of documents are duplicates
- We trace how these data quality failures shaped architectural decisions, from bypassing encounter-based scoping to using agentic search over fixed retrieval pipelines
The system runs entirely on-premise (Qwen 3.6 35B on 4xH100), with every extracted value grounded in cited source passages for clinician verification.
We'd love to hear from others working on clinical IE in production, especially around data quality challenges in real hospital systems.
Neat paper. It is refreshing to see a focus on the metadata gap in clinical data, as standard RAG pipelines really do struggle when patient records are this fragmented and messy. Using an on-premise agentic approach to force grounding for clinician verification seems like a sensible way to handle the trust issue.
I am curious, given the 96.5% acceptance rate, what were the most common reasons for the 3.5% of extractions that clinicians rejected?
I made a podcast on it with ResearchPod, it makes it easy to get the key concepts on the go:
https://researchpod.app/episode/51b990fd-b654-4e53-9247-ed076590c9b8
Thanks for the kind words and the podcast!
To your question: of the 253 rejections, the vast majority (87.4%) were values that needed correction rather than wholesale replacement. The errors concentrate heavily in dates and treatment timelines (10 of 74 fields account for 72% of all rejections), which makes sense given how temporal information is scattered across documents with unreliable timestamps. We break this down further in the error analysis appendix (Tables 7-9).
Get this paper in your agent:
hf papers read 2606.19602 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper