AI risk management in clinical settings

The range extends from systems that flag abnormalities in radiology and move urgent cases up the worklist, through early warning scores in intensive care units, analyses in laboratory medicine and pathology, and dosing suggestions, to language models that draft discharge letters or document consultations. Then there are tools without direct patient contact, for example for staff rostering, bed management or appointment scheduling. The closer a system operates to diagnostic or therapeutic decisions, the more directly its errors affect people.

An update nobody noticed, a dataset in which certain patient groups barely appear, or a shifted calibration can be enough to change findings – often without anyone on the ward noticing. For deployers, the decisive question is therefore not whether a system is “intelligent”, but which decision it influences, who relies on its outputs and what happens if it is wrong or fails.

Three groups of risk

• Technical risks: misclassifications, declining model performance, attacks on data, model or interfaces, and service outages.
• Organisational risks: unclear responsibilities, lack of training, unmanaged updates, use of unapproved AI tools and gaps in incident handling.
• Ethical and legal risks: disadvantaging certain patient groups, lack of traceability, data protection breaches and a loss of trust between clinicians and patients.

Two terms you need to know

Model drift refers to the gradual loss of a model’s performance because input data, patient population, devices or clinical workflows have changed compared with the conditions under which it was validated. The MDCG 2025-6 guidance recommends designing datasets so that unintended bias and dataset drift remain manageable and subgroups in which a model performs worse, including over time, can be identified.

Automation bias describes the tendency to rely on machine outputs or to place too much trust in them. The AI Act requires that the people assigned to oversight are enabled to recognise this tendency, interpret outputs correctly, disregard them in individual cases and halt the system using a stop button or a similar procedure.

This page summarises the German-language expert article “Risikomanagement für KI-Systeme im Medizinbetrieb” (risk management for AI systems in clinical settings) from mt|medizintechnik, issue 2.2026, restructures it and adds developments up to October 2026, in particular the Digital Omnibus on the AI Act.

The AI Act does not classify medical AI as high-risk across the board. The decisive route runs through product legislation: if the AI is itself a medical device or in vitro diagnostic device, or a safety component of one, and a notified body must be involved in the conformity assessment, it counts as a high-risk AI system. Because the MDR usually places software that provides information for diagnostic or therapeutic decisions at least in class IIa, this applies to most diagnostic and therapeutic AI applications. The AI Act requirements are then assessed as part of the notified body’s existing MDR procedure – there is no second certification.

In addition, the AI Act directly designates a number of use cases in the health sector as high-risk: evaluating emergency calls and triaging emergency patients, decisions by public authorities on access to public healthcare services, and risk assessment and pricing in health and life insurance. If such a triage system is also a medical device assessed by a notified body, the product legislation route takes precedence.

Where NIS2 and data protection come in

For hospitals that the BSI Act covers as particularly important or important entities, AI systems are part of the IT that must be protected: a significant security incident in an AI service must be reported early – an early warning within 24 hours at the latest, an incident notification with an initial assessment within 72 hours and a final report no later than one month after that. Data protection law requires a data protection impact assessment for high-risk processing; the AI Act obliges deployers to use the information supplied by the provider for this purpose.

Timeline from the deployer’s perspective

• Since 2 February 2025: prohibitions on certain AI practices; obligation to promote staff AI literacy – since the Omnibus, explicitly without having to guarantee a particular level of competence.
• Since 2 August 2026: general application of the AI Act and transparency obligations; supervision by the AI Office and national authorities.
• Since 11 September 2026: manufacturers’ reporting obligations under the Cyber Resilience Act for actively exploited vulnerabilities and severe security incidents.
• From 2 December 2027: high-risk obligations for directly listed use cases, such as emergency triage, where the system is not assessed as a medical device by a notified body.
• From 2 August 2028: high-risk obligations for AI medical devices – for providers and for deployers. Systems placed on the market earlier are only covered after significant design changes; systems intended for use by public authorities must comply by 2 August 2030 at the latest.

The Omnibus also contains an opening clause whose outcome is still uncertain: for products governed by their own sector legislation – including medical devices – the Commission may limit individual AI Act requirements by delegated act, provided the sector legislation offers equivalent protection. It has until 2 August 2027 to do so. This changes little for hospital planning, as deployer obligations remain unaffected. A Commission proposal from December 2025 to revise the MDR and IVDR goes further: it would move AI medical devices into a lighter regulatory group within the AI Act, so that the AI requirements would be implemented primarily through the MDR. It has not been adopted; until it is, the dates set out here apply.

A provider is anyone who develops an AI system, or has one developed, and places it on the market or puts it into service under their own name; for medical devices, this is almost always the manufacturer under the MDR. A deployer is anyone who uses the system professionally under their own authority. The MDCG 2025-6 guidance makes clear that the deployer is not the same as the “user” under the MDR: in a hospital, the organisation is the deployer, while the physician at the reporting workstation is the user.

When the organisation itself becomes a provider

• It markets a high-risk system that is already on the market under its own name or trademark.
• It substantially modifies a high-risk system, meaning in a way that was not foreseen in the original conformity assessment and that affects compliance or the intended purpose.
• It uses a system that was not previously high-risk – including a general-purpose language model – for a purpose that makes it high-risk.

According to MDCG 2025-6, a substantial modification requires a new conformity assessment even if the organisation only continues to use the modified system itself. Changes predetermined and assessed by the manufacturer, for example in a predetermined change control plan, are not considered substantial. In-house developments that a healthcare institution uses exclusively internally under the rules for in-house devices are not high-risk AI according to the guidance; other AI Act obligations still apply.

What belongs in the contract

• Information on data provenance, validation populations and known performance limits.
• An update policy with advance notice periods, release notes and a rollback option.
• Access to logs and support with incident analysis and the data protection impact assessment.
• Reporting channels, response times and contacts for security matters.
• Security commitments for cloud components and third-party libraries, including a software bill of materials.
• End of support and the orderly handover or deletion of data when the contract ends.

The AI Act requires only the provider to maintain a dedicated risk management system. Deployers have obligations relating to monitoring, oversight, data, logs and reporting. Structured risk management within the organisation is nevertheless the practical basis for meeting these obligations. The operator standard IEC 80001-1 describes exactly this approach for connected medical devices and health software: risk management before, during and after integration into the hospital IT.

1. 1
   Before procurement: purpose and classification

   Clarify the clinical benefit, intended purpose, the organisation’s role and the regulatory classification. Define acceptance criteria: which error rates and which residual risks are acceptable for this use, and who decides?

2. 2
   Before go-live: testing in your own environment

   Use your own representative data to check whether the performance stated by the manufacturer is achieved in your patient population. Secure interfaces, permissions and the network segment, appoint and train oversight personnel, and define the fallback process.

3. 3
   In operation: monitoring and feedback

   Continuously track performance indicators, override rates, data quality and availability. Document anomalies, report them to the provider and suspend use if there is a risk.

4. 4
   When things change: controlled approval

   Every model update, every new interface and every new area of use goes through assessment, testing, approval and documentation. Check whether the change was predetermined by the manufacturer or would push the organisation into the provider role.

5. 5
   At decommissioning: an orderly exit

   Ensure a replacement process, secure logs and evidence in line with retention periods, close accounts and interfaces, have the provider delete the data and feed the lessons learned into the next procurement.

STRIDE goes back to Loren Kohnfelder and Praerit Garg, who described the scheme at Microsoft in 1999. It assigns threats to six categories and can be applied well to a data flow diagram: for each component and each connection, you ask which category is conceivable there. For an AI diagnostic system, this typically includes the imaging device (called a “modality” in radiology), the image archive, the inference service running on premises or in the cloud, the results display, the hospital information system and the manufacturer’s update channel.

AI-specific threats

The AI Act itself names typical attacks on AI: data and model poisoning, deliberately manipulated inputs designed to deceive a model, attacks on confidentiality and the exploitation of model flaws. For applications using language models, the 2026 edition of the OWASP Top 10 for LLM Applications provides an up-to-date structure. For healthcare organisations, the following patterns are particularly relevant:

A matrix with two axes has proven effective. One axis represents likelihood, informed by technical factors such as attack surface and model robustness, organisational factors such as training level and process maturity, and human factors such as time pressure and automation bias. The other represents the severity of the consequences – medical, in terms of data protection, operational and with regard to trust in the organisation.

Testing specifically for bias

Bias arises when training data does not reflect the organisation’s own patient population or when workflows and devices change. Deployers should therefore evaluate performance indicators not only overall but also by relevant subgroups, insofar as the data and the legal basis allow. Since the Omnibus, the AI Act permits deployers of high-risk systems, by way of exception, to process special categories of personal data such as health data in order to detect bias – subject to strict safeguards and without this creating an obligation.

Fundamental rights impact assessment: when it is mandatory

The AI Act requires a fundamental rights impact assessment only for the high-risk use cases it lists itself, and only from public bodies and private providers of public services, as well as from deployers that assess creditworthiness or price life and health insurance. Public hospitals and private hospitals with a public healthcare mandate may therefore be affected when they deploy a triage system in the emergency department, for example; if that system is also a medical device assessed by a notified body, the precedence of product legislation means it should be checked case by case whether the obligation applies. It does not apply to AI medical devices that are high-risk solely via the MDR. The Omnibus allows deployers to refer to a data protection impact assessment that has already been carried out.

1. 1
   Prevention

   Security starts with selection: demand secure by design from the provider, validate with representative test data of your own, and implement network segmentation, multi-factor authentication and role-based permissions. Getting a second, independent opinion before go-live – from an external security assessment or a partner hospital with experience in the same field – reveals blind spots before they affect patients.

2. 2
   Detection

   Spot deviations early: monitor performance indicators and input data for drift, analyse logs, define alerts for unusual behaviour and use overrides by staff as an early indicator. In intensive care and the emergency department, what counts is how quickly an anomaly reaches the responsible team.

3. 3
   Correction

   When something goes wrong, people keep the final say at critical points and can stop the system. Fallback processes without AI, the ability to roll back to an earlier model version, clear escalation paths and a rehearsed incident handling procedure limit the damage.

4. 4
   Evidence

   Documentation ties everything together: classification, risk assessment, test results, approvals, training, overrides, incidents and corrections. It demonstrates to supervisory authorities and auditors that obligations are being met and is the basis for continuous improvement.

Reporting when it gets serious

If the organisation has reason to believe that using the system presents a risk, it informs the provider or distributor and the market surveillance authority without undue delay and suspends use. In the event of a serious incident, the AI Act requires the deployer to inform the provider first, without undue delay. For AI medical devices, health-related incidents are handled through the MDR vigilance system – depending on severity, the manufacturer reports within 15, 10 or 2 days at the latest. In Germany, operators and users must also report suspected serious incidents without delay to the competent federal authority, usually the BfArM. For these devices, only infringements of obligations that protect fundamental rights are reported under the AI Act. Reports under the BSI Act and data protection law may be required in parallel.

Concrete controls at a glance

• An inventory of all AI systems with purpose, classification, owners and data flows.
• Named oversight personnel for each system, with documented training and the authority to override.
• Logging with model version and user action, retained for at least six months.
• An approval process for updates, including testing in a reference environment.
• Regular security testing and AI red teaming, especially for language models and agents.
• An agreed reporting matrix covering the provider, medical device vigilance, the BSI and the data protection authority.

Many organisations already operate an information security management system (ISMS), a quality management system and data protection management. AI risks can be integrated into these: as a separate risk category, with dedicated controls and a clear role in approving new systems. The overview below shows what each standard contributes.

For hospitals covered by the BSI Act, embedding also makes legal sense: the required risk management measures include incident handling, supply chain security and security in acquisition, development and maintenance, including the handling of vulnerabilities. AI services and their providers therefore belong in the supplier and vulnerability management of the ISMS.

1. 1
   Days 1–15: assign responsibility

   Designate a function responsible for AI, bring clinical engineering, IT security, data protection, quality management and medical leadership together in a committee, and make the approval of new AI systems subject to this committee.

2. 2
   Days 16–30: build the inventory

   Record all AI systems, including embedded functions in devices and cloud services. For each system, document the purpose, provider, data flows, user groups and regulatory classification.

3. 3
   Days 31–45: prioritise risks

   For the systems with the strongest patient relevance, create a STRIDE threat model and assess the impact. The result is a list of the most urgent measures.

4. 4
   Days 46–60: contracts and providers

   Review existing contracts for update policy, log access, data provenance, reporting channels and end of support; renegotiate missing commitments and define procurement requirements for future purchases.

5. 5
   Days 61–75: oversight and AI literacy

   Appoint oversight personnel for each system, deliver training tailored to each target group on performance limits and automation bias, and inform staff and their representatives.

6. 6
   Days 76–90: embed in operations

   Set up monitoring and log retention, put the change procedure and reporting matrix into effect, run an incident handling exercise and adopt the plan for the following twelve months.

The plan is intended as guidance and does not replace an assessment of the individual case. Scope and sequence depend on the size, care mandate and AI portfolio of the organisation.