Healthcare AI toolkit: what the audit found, and how we fixed it
We ran our audit tool on a healthcare AI toolkit. It is a library that teams build clinical-adjacent applications on. This case study sets out what the audit found, what it means for a team that builds on the toolkit, and what our forward-deployed engineers did after the report.
- AI components detected 5
- Findings 5 All medium
- Tier High-risk
- Obligations checked 28
At a glance
- Sector
- Healthcare software and clinical-adjacent tooling
- System
- A Python toolkit for healthcare AI. It includes examples for discharge summaries, question answering over clinical records and clinical coding.
- EU AI Act scope*
- The tool places the toolkit in the high-risk tier. The tier depends on the intended use of each application built on the toolkit.
- AI components detected
- 5: three AI SDK components (LangChain core, LangChain, Hugging Face transformers) and two prompt templates
- Findings
- 5, all medium
- Tier
- High-risk
- Obligations checked
- 28 provider obligations
- Stack detected
- LangChain · Hugging Face transformers · prompt templates
Context
A toolkit that other teams build on carries a specific risk.
Its examples can be copied into clinical software, and its prompts can end up in front of patients and clinicians. The EU AI Act treats many AI systems in healthcare as high-risk, and the duties depend on what each application is for.
Teams building on a healthcare toolkit ask four questions:
- What is the toolkit for, and what must it never be used for?
- Who is responsible for each application built on it?
- Which prompts and models are in use, and who owns them?
- What evidence exists for each release?
What we audited
Our audit tool reads the code and the documentation.
It lists the AI components and prompts, places the use in an EU AI Act tier, and checks the evidence against a register of provider obligations. Each finding is tied to an article or control and given a severity.
AI inventory
The SDKs, models and prompt templates in the code and the examples.
Risk tier
Whether the use is high-risk, and what decides it.
Obligations
The 28 provider obligations in the register, checked against the code, the documentation and the logging the toolkit provides.
Next steps
Items that need a running system or an interview, such as incident handling and how teams use the examples.
What the audit found
-
MEDIUMCookbook example
AI SDK detected: LangChain core
Why it mattersThe toolkit's prompt chain depends on this library. Changes to it change the outputs.
-
MEDIUMDischarge summary example
AI SDK detected: Hugging Face transformers
Why it mattersSummaries are generated by a model. The model version and data handling must be recorded.
-
MEDIUMConfiguration reference
AI SDK detected: LangChain
Why it mattersConfiguration sets which model and chain run. It should be versioned.
-
MEDIUMQuestion-and-answer example
Prompt template detected: inline system prompt
Why it mattersA prompt in the documentation is copied into applications. Its owner and version must be clear.
-
MEDIUMClinical data mapping configuration
Prompt template detected: system mapping
Why it mattersMappings decide how clinical data is read. They need an owner and a review.
-
TIERAudit summary
Risk tier: high-risk
Why it mattersDepends on intended use. Brings duties on risk management, documentation, records and oversight.
Across the 28 obligations, the review found evidence in full for 1, in part for 12 and none for 4. 7 need a runtime review, and 4 do not apply.
Some strengths were recorded as well. The toolkit states clearly what data it does and does not send, it includes authentication and encrypted connections, and its developer documentation is extensive.
What we did next
After the report, our forward-deployed engineers either fixed the issues hands-on or advised and worked alongside the team to close them.
The team that builds on the toolkit makes the decisions about its use. Our engineers worked on the parts that are code and configuration, and advised on the parts that are policy.
-
Fixed hands-on
Our engineers moved the prompts and the clinical mapping into reviewed, versioned configuration with a named owner. They added AI event logging next to the existing web request log, so each model call records its input reference, output reference and version.
-
Advised and worked alongside the team
Our engineers helped the team write down the intended use of each application, and to record whether the output advises a clinician or decides something. The team owns that decision. They also set up the risk management file, the technical documentation set and a route for serious incidents, with clinical safety and legal staff.
What we would advise a team building on this toolkit
- Write the intended use of each application before any build starts. The tier follows from it.
- Record whether the output advises a clinician or decides something.
- Version every prompt and mapping, and give each one an owner.
- Log AI events, not only web requests.
- Keep a risk file and a technical file for each release, and a route for serious incidents.
What the team now has
A written intended use.
Each application states what the output is for and who uses it.
Prompts and mappings under control.
Each one has a version and an owner.
AI event logs.
Each model call can be traced to its input reference, output reference and version.
A risk file and a technical file.
Both are set up for each release, and a person signs them off.
A route for serious incidents.
Reports go to a named owner, with corrective action recorded.
How it is built
The toolkit is a library. Teams import it, connect it to records, and build an application on top. The diagram below shows the design the audit pointed towards. The team that builds on the toolkit owns the application and its tier.
Questions teams ask us
Talk through a system like this
Bring your application, your intended use and your clinical review process. We will run the same audit tool on it and walk through the findings, the fixes and the handover pack with you.
Read on: Credit decision support: what the audit found, and how we fixed it · An applicant tracking system: what the audit found, and how we fixed it
* EU AI Act scope: the audit tool places the toolkit in the high-risk tier. The tier of each application depends on its intended use and its role in care.
