I started noticing the same pattern in our usability tests: a confident suggestion on screen, a tired clinician at 03:00, and a near-miss that never made it into the incident log because everyone assumed "the machine is right". To be fair, the intention behind AI support is to reduce errors — granted — but if your UI is designed so the clinician cannot reasonably question or override the AI, that design choice is itself a risk control (and a brittle one).
This is not academic. IEC 62366-1 (usability engineering) expects manufacturers to identify use-related hazards and verify that the user interface mitigates them. The EU AI Act (Article 14) requires appropriate human oversight measures for high‑risk systems. ISO 14971 asks us to treat use-related hazards like any other hazard. In practice this means automation bias — users deferring to automated suggestions — should be in your risk analysis, your usability report, and your human‑oversight strategy in the Technical File.
What automation bias looks like in the wild
I've seen it in three recurring flavours:
- Silent confidence: the UI shows a single recommended action without uncertainty or provenance. Clinician accepts without checking data.
- Frictionless override buried: an override exists but is tucked behind multiple clicks; in an emergency people default to the path of least resistance.
- Audit blindness: the system logs the AI action but not the human override rationale, so post‑market surveillance cannot reconstruct why someone overrode or didn't.
Each is a design decision with regulatory consequences.
Why this is regulatory, not just UX
- IEC 62366‑1: you must identify use‑related hazards and mitigate them through design and verification. If your UI increases the likelihood of an incorrect action because users over‑trust the AI, that's a use‑related hazard and will be a usability finding.
- ISO 14971: automation bias is a foreseeable source of harm; it needs classification, risk estimation, and documented risk control measures. You cannot just write "train users" as the only control and call it a day.
- EU AI Act, Article 14: for high‑risk AI, you must implement human oversight measures that reduce risks linked to automation bias (e.g., ensure the human can understand and reasonably contest the AI output). The AI Act does not leave "human oversight" vague.
Notified bodies are increasingly flagging both IEC 62366 non-conformities and weak human oversight measures in AI‑assisted devices. In audits I've run, a lack of active push‑back mechanisms in the UI was the single most common finding.
Practical risk controls that actually work
In my teams we moved from "UX is pretty" to "UX is a safety subsystem". Here are controls that became standard in our technical files and usability reports:
- Explicit uncertainty and provenance
- Show confidence bands, key inputs, and the most recent data timestamp. If the user cannot see these fast, they tend to accept the output.
- Reduce passive acceptance
- Require an explicit, deliberate confirmation for high‑risk recommendations (not a single tap). In practice this means a two-step confirm or a short forced pause for critical actions.
- Make override immediate and low-friction
- Place the override next to the recommendation. Log the override and capture a short, structured rationale (predefined options + optional free text).
- Scenario-based verification
- Run summative usability tests under realistic cognitive load (interruptions, fatigue, incomplete data). Measure reliance and record near-misses.
- Traceable human oversight strategy
- Include a documented human‑oversight concept in the Technical File: who may override, when, what training is required, and how overrides are logged and reviewed.
- Post‑market monitoring for automation bias
- Add specific PMCF questions and PSUR signals that look for patterns of over‑reliance or frequent overrides without rationale.
How to show this to a notified body
Bring evidence, not assertions. Auditors want to see:
- The hazard analysis with automation bias explicitly listed (ISO 14971)
- Usability tests that mimic real-world distractions and quantify over‑reliance (IEC 62366‑1)
- The human oversight strategy mapped to the AI Act Article 14 elements: operator competences, HMI design, and logging/auditability
- Traceability: from hazard → risk control → verification method → test results → change control tickets
In practice this means traceability matrices need to be living artifacts, not dead Word tables. Use your eQMS so CAPAs, changes and verification evidence are linked and reviewable.
Where teams trip up
- "Train them better" as a solo control. Training helps, but it's not a design control. It’s compensatory and must be combined with interface and system controls.
- Hiding uncertainty because it "confuses clinicians". This choice shifts the burden onto the user and increases risk.
- Expecting clinical governance alone to catch it. If the UI invites blind trust, the clinician will accept it under workload.
Quick checklist you can use today
- Is automation bias listed in your ISO 14971 risk analysis? (If not, add it.)
- Does your summative usability test include time pressure and interruptions?
- Can the clinician override without friction and is the override logged with rationale?
- Do your logs capture AI inputs, AI output confidence, timestamp and the human response?
- Is there a defined human‑oversight strategy in the Technical File linked to verification evidence?
Final note
This is not about demonising AI; it's about recognising that when you put a suggestion in front of a human, you've created a small socio‑technical system. The interface is not neutral — it is a safety device. Treat it with the same rigour you treat hardware interlocks.
How have you designed a UI to reduce automation bias in your device, and what evidence did your notified body find convincing?
