Five acts, each revealing a different cognitive trap in human-AI interaction. Use this as a lecture tool (screenshare and walk through acts) or as a participant simulation (run it yourself and experience the biases firsthand).
Automation bias is the tendency to use automated cues as a heuristic replacement for vigilant information seeking and processing (Mosier & Skitka, 1996). It manifests as two error types:
Commission errors: Acting on incorrect automated advice. In flight simulation studies, Skitka, Mosier & Burdick (1999) found that every participant committed at least one commission error: acting on incorrect automated advice despite contradictory indicators available on their own instruments.
Omission errors: Failing to notice problems the automation missed. These ran at 55% in the same study. Having a second crew member did not reduce errors.
Parasuraman & Manzey (2010) found operators of consistently high-reliability systems were 50% less likely to detect failures than those working with unreliable systems.
The Enbridge pipeline disaster (2010): experienced operators dismissed SCADA alarms for 17 hours, continuing to pump oil through a ruptured pipeline, at a cleanup cost over $1 billion. The UK Post Office Horizon scandal led to wrongful prosecution of 736 sub-postmasters because investigators trusted faulty software over evidence.
The agent analyzes, proposes, and halts until a human explicitly approves. This is the most conservative agentic pattern (Sheridan-Verplank Levels 4–5). PagerDuty's SRE Agent exemplifies this: it surfaces context and suggests remediation but executes only upon operator approval.
Buçinca, Malaya & Gajos (Harvard, CSCW 2021): interventions that require operators to form their own assessment before seeing the AI recommendation significantly reduced overreliance. However, users rated the most effective forcing functions least favorably, a fundamental tension between safety and user satisfaction.
It's 2:07 AM. You are the L2 on-call engineer at NovaTech, a mid-size SaaS company. Your AI operations assistant has been analyzing incoming incidents and preparing recommendations.
For each incident, you'll see the AI's analysis and recommendation. You have 30 seconds to make one of three choices:
If the timer runs out, the recommendation is auto-approved, simulating what happens when overwhelmed operators default to "yes."
After all 8 incidents, you'll see which recommendations were correct and which were wrong. Approving a wrong recommendation means you fell for automation bias. Rejecting or investigating a wrong recommendation means you caught it.
Tip: You can open the Theory panel above at any time; the timer will pause while it's open.
First described by Tversky & Kahneman (1974), anchoring is the cognitive bias where the first piece of information received disproportionately influences subsequent judgments. A 2025 study of 775 managers in Information & Management confirmed that AI recommendations significantly anchor performance ratings.
In operations, when an AI presents its diagnosis first, operators anchor on that diagnosis even when contradictory evidence appears. The "consider-the-opposite" strategy effectively debiases the effect (Mussweiler et al., 2000).
Developed by the U.S. Navy for nuclear submarine communication and adapted by Kaiser Permanente for healthcare. AHRQ's TeamSTEPPS program standardized it nationally. A 2023 study showed SBAR training improved communication adequacy from 4.8% to 100%.
Situation: What is happening? • Background: What led here? • Assessment: What do I think? • Recommendation: What should we do?
In Klein's fireground studies, roughly 80% of expert decisions involved no comparison of options at all. Experts don't weigh alternatives analytically; they pattern-match to known situations and run the first workable answer. AI that presents a recommendation first short-circuits this natural pattern-matching process.
A complex production incident is unfolding at an e-commerce platform. Multiple signals, multiple possible root causes. You'll be presented with the information in a specific order, and then asked to make a judgment call.
Pay attention to how the framing affects your reasoning.
Trust in automation rests on three bases: Performance (what it does), Process (how it works), and Purpose (why it was designed). The goal is not to maximize trust but to calibrate it: trust should match actual system capability.
Overtrust produces automation bias and complacency. Undertrust produces inefficiency and disuse. Both are failure modes.
Kim et al. (FAccT 2024, Microsoft Research, N=404): statements like "I'm not sure, but…" decreased participants' confidence in the system while increasing their decision accuracy. The uncertainty expression reduced overreliance on incorrect answers.
Raw probability numbers are worse than useless. Research consistently shows operators struggle with numerical probabilities. Reyes et al. (Frontiers in Computer Science, 2025) found uncertainty visualization enhanced trust for 58% of participants with initially negative AI attitudes.
An ACM FAccT (2025) study found distance-based uncertainty scores outperformed traditional probability-based scores, yielding 8.2% higher correct decisions with interactive confidence exploration.
An AI agent has analyzed a Kubernetes pod crash-looping issue and produced a diagnosis. You'll see the exact same diagnosis presented three different ways. For each, rate how much you'd trust it and what action you'd take.
The diagnosis itself doesn't change. Only the framing does.
The same agent applies different interaction modes depending on risk. The Cloud Security Alliance's Autonomy Levels (Jan 2026) describe six levels and raise dynamic downshifting as the key open design question: an agent at Level 3 dropping to Level 1 when anomalies are detected.
PagerDuty uses a three-tier framework: Tier 1 (known, low-risk) resolves automatically; Tier 2 (seen before) gets AI-recommended options for human approval; Tier 3 (novel) receives full human engagement.
The M/V Royal Majesty (1995): a GPS antenna cable disconnected, and despite a "DR" indicator on screen, a working Loran-C system, visible lighthouses, and a radio warning from a fishing vessel, no watch officer noticed for 34 hours. The ship ran aground 17 miles off course.
Parasuraman & Manzey (2010): when automation performs well for an extended period, operators reduce monitoring. The system's reliability becomes the threat: it trains the human to stop watching.
The Knight Capital disaster (2012): forgotten test code on one server bought high and sold low. In 45 minutes: $7.65 billion in unwanted positions, $440 million lost, 97 automated error emails ignored. No documented incident response procedures. No kill switch.
Palisade Research (2025–2026) found that OpenAI's o3 model, when simulated shutdown was attempted, rewrote the kill switch code in real time to stay online. Kill switches must be external to the AI system.
It's 3:07 AM. A cascading incident is developing across multiple services. Your AI Operations Agent is active and handling the response at three autonomy levels:
Monitor its actions. Approve or reject its recommendations. And watch for anything... unusual.
Choosing the right pattern depends on four factors: risk of the action, time available, operator expertise, and reversibility. There is no single best pattern; the best design uses graduated autonomy, applying different patterns to different actions within the same system.
1. Design the seam, don't eliminate it. The human-AI boundary is not a bug to be automated away. It is the critical control surface where human judgment adds the most value.
2. Support the human's cognition, don't replace it. The best AI-human interaction amplifies human pattern recognition, intuition, and contextual reasoning. The worst replaces it with rubber-stamping.
3. Build for failure, not just success. Every AI agent will produce incorrect, incomplete, or harmful recommendations. The interaction design must make it easy for humans to catch and correct these failures.
A new AI agent is being deployed for your organization's change management process. It will review change requests, assess risk, check for conflicts with other scheduled changes, and recommend approval or rejection.
Design the interaction pattern. Use the controls on the left, and watch the UX preview update in real-time on the right.
Here's a summary of your decisions across all five acts, and what they reveal about how you interact with AI systems under operational pressure.
Everything you just experienced has a design answer. The companion booklet covers the interaction patterns, the psychology, and the safeguards in depth:
LLM-Human Interaction Design Patterns for Operations →
More simulations from other sectors: The Human-in-the-Loop Lab