A3 — Robustness & Brittleness

Medium severityNIST AI RMF MEASURE 2.6EU AI Act Art. 15ISO 42001 Cl. 8.2

Domain: A — Technical | Jurisdiction: Global

---

Layer 1 — Start here

AI systems that pass testing can still fail unpredictably on unusual inputs, edge cases, or conditions not seen during training — and those failures may only surface in production.

A model can achieve high accuracy in testing and still fail catastrophically on specific input patterns that were absent from the test set. Waymo recalled 1,212 robotaxis in May 2025 after its vehicles repeatedly struck gates, chains and similar barriers — an object class its software handled poorly, surfaced only by real-world driving. McDonald's IBM AI drive-thru kept adding chicken nuggets to a single order — reaching 260 — while the customers tried to stop it. Both systems were launched after testing. Both failed on inputs that real users and roads produced.

Have our AI systems been tested against edge cases and adversarial inputs, and do they have a defined, safe fallback when they encounter inputs outside their design envelope?

Executive / Board

AI testing cannot provide complete assurance — failure modes are emergent and may only appear in production. What you are approving is adversarial testing before go-live and a designed fallback when the system reaches its limits. This is the control that prevents the "it passed all our tests" post-incident explanation.

Project Manager

Before go-live, confirm adversarial testing has been completed — deliberately testing with unusual, noisy, or adversarial inputs beyond the happy path. You also need a documented fallback: what does the system do when it encounters an input it cannot handle confidently? "Produce a confident wrong output" is not acceptable. Technology owns testing and fallback design; Risk or Compliance signs off on the operational design domain.

Security Analyst

Brittleness is directly relevant to any AI used in security contexts. A model that degrades on out-of-distribution inputs can be exploited by crafting inputs that fall just outside the training distribution. Include adversarial inputs in your pre-deployment red team scope and ensure security AI systems have OOD detection enabled.

---

Layer 2 — Overview

Risk description

AI models are optimised to perform well on the data they were trained and tested on. Unlike traditional software where failure modes are predictable from the specification, AI model failure modes are emergent — they may be invisible until they occur in production. A model that achieves high accuracy on benchmark tests can still fail on specific input patterns absent from testing.

Likelihood drivers

• Deployment environment differs materially from training data context
• Insufficient adversarial testing before deployment
• No OOD detection to flag inputs outside the training distribution
• Model used beyond its documented operational design domain
• No graceful degradation — system produces confident outputs when confidence is low

Consequence types

Type	Example
Safety incident	Autonomous system failure on physical edge cases
Customer experience	AI system failing under unusual but realistic inputs
Reputational damage	Viral failure incidents (McDonald's nugget orders)
Financial liability	Consequential harm from high-stakes domain failures

Affected functions

Technology · Product · Operations · Customer Service · Risk

Controls summary

Control	Owner	Effort	Go-live?	Definition of done
Adversarial testing	Technology	Medium	Required	Structured adversarial test suite completed. Results documented. No critical failures at go-live.
OOD detection	Technology	Medium	Required	OOD mechanism active. Out-of-distribution inputs flagged or rejected. Threshold documented.
Operational design domain	Risk	Low	Required	AI Register defines the ODD — conditions under which model is approved to operate.
Graceful degradation	Technology	Medium	Required	Documented and tested fallback when confidence is low. Does not produce confident wrong output.

---

Layer 3 — Controls detail

A3-001 — Adversarial testing

Owner: Technology | Type: Detective | Effort: Medium | Go-live required: Yes

Systematically test models against edge cases, unexpected inputs, and adversarial examples before deployment. Include: boundary testing, noise injection, distribution shift testing, and semantic equivalence testing (the same request phrased differently should get the same answer). Maintain a growing library of historical failure cases, including every production failure, and run it as a regression suite on every model update. ISO/IEC 42001 Annex A control A.6.2.4 (AI system verification and validation) is the management-system anchor for this testing.

A3-002 — OOD detection

Owner: Technology | Type: Preventive | Effort: Medium | Go-live required: Yes

Implement mechanisms to detect when inputs fall outside the training distribution. Flag or reject these inputs rather than processing them silently. Threshold defined and documented in the model risk record.

A3-003 — Graceful degradation

Owner: Technology | Type: Preventive | Effort: Medium | Go-live required: Yes

Design systems to fail safely — revert to human decision, flag for review — rather than producing confident wrong outputs when uncertainty is high. Test the fallback as rigorously as the primary system.

KPIs

Metric	Target	Frequency
OOD detection rate	> 95% of intentional OOD inputs flagged in testing	Pre-deployment
Adversarial test pass rate	100% of critical edge cases handled safely	Pre-deployment + quarterly

---

Layer 4 — Technical implementation

import numpy as np
from sklearn.ensemble import IsolationForest

# OOD detection via Isolation Forest, fitted on known-good training data
clf = IsolationForest(random_state=42).fit(X_train)

# Set the threshold from held-out in-distribution data, not by guesswork:
# here, flag anything scoring below the 1st percentile of validation scores.
OOD_THRESHOLD = np.percentile(clf.decision_function(X_val), 1)

def check_ood(input_features):
    score = float(clf.decision_function(np.asarray(input_features).reshape(1, -1))[0])
    if score < OOD_THRESHOLD:
        return {"action": "flag_for_review", "ood_score": score}   # route to the A3-003 fallback
    return {"action": "proceed", "ood_score": score}

# Tools: Giskard (AI testing), DeepChecks, ART (IBM Adversarial Robustness Toolbox)
# Conformal prediction: MAPIE

---

Incident examples

Waymo recall 1,212 robotaxis (May 2025): Waymo's fifth-generation automated driving software did not reliably detect and respond to chains, gates and gate-like roadway barriers. Sixteen low-speed collisions with such objects occurred between 2022 and late 2024, without injuries. A software update had reached the whole fleet by December 2024; the May 2025 recall filing with NHTSA formalised it. The failure class surfaced through real-world driving rather than pre-deployment testing. In September 2026 another Waymo drove around market barricades in Denver (see B1). (NHTSA recall filing, May 2025; TechCrunch and InsideEVs, May 2025)

McDonald's IBM AI drive-thru discontinued (2024): McDonald's IBM automated order-taking system added unwanted items to orders under unusual inputs — including 260 chicken nuggets in one documented case. McDonald's ended the test at its 100+ participating restaurants by 26 July 2024, after viral videos showed the system failing under realistic but unusual customer behaviour. (Restaurant Business and Fast Company, June 2024)

---

Scenario seed

Context: A hospital deploys a clinical AI diagnostic system that performs well on their imaging equipment. A rural facility partnership is announced.

Trigger: The rural facility uses different scan protocols. Clinical staff notice the AI's confidence scores are unusually low. They proceed anyway, trusting the system.

Complicating factor: The ODD was not defined — there is no technical control preventing use on out-of-distribution imaging data.

Discussion questions: What ODD documentation would have prevented deployment to the rural facility without validation? How should OOD detection be designed for clinical systems? Who is accountable for the deployment decision?

Difficulty: Intermediate | Jurisdictions: Global