A2 — Model Drift / Performance Degradation

Medium severityISO 42001 Cl. 9.1EU AI Act Art. 72APRA CPS 230

Domain: A — Technical | Jurisdiction: Global

---

Layer 1 — Start here

AI model performance degrades over time as the real world shifts away from training conditions — silently, without visible error, until the consequences become apparent downstream.

Unlike traditional software, AI models can become less accurate over time as the world changes around them. Models trained on pre-pandemic behaviour broke when spending patterns shifted in 2020. A hosted language model can change underneath you when the vendor updates it. The model does not alert you — it continues producing outputs that are increasingly wrong.

Can we confirm that every deployed AI model has active performance monitoring and defined thresholds that trigger review or retraining?

Executive / Board

If your organisation makes decisions based on AI models — credit, fraud, underwriting, claims — and those models are not actively monitored, you may be acting on increasingly unreliable outputs without knowing it. The audit finding means your monitoring framework is insufficient. Approving remediation means investing in monitoring infrastructure that tells you when a model is no longer performing as expected.

Project Manager

Before going live with any predictive AI model, confirm that a monitoring framework will be active from day one. You need sign-off that drift detection is configured, performance thresholds are defined, retraining triggers are documented, and a named owner is responsible for acting when thresholds are breached. These are Technology deliverables with Risk oversight.

Security Analyst

Model drift affects you when security AI systems — SIEM anomaly detection, fraud scoring, access risk models — degrade silently. A degraded fraud model that misses attack patterns is a security control failure. Ensure security AI systems have defined performance thresholds and retraining triggers, and that degradation alerts route to security.

---

Layer 2 — Overview

Risk description

AI models are trained at a point in time on data reflecting the world as it was. Three mechanisms drive degradation: data drift (input distribution changes), concept drift (the relationship the model learned no longer holds), and model decay (representations become stale without identifiable input changes). Because models rarely fail visibly when drifting, degradation can persist undetected for months.

Likelihood drivers

• No monitoring framework in place post-deployment
• No scheduled review dates in the AI Register
• Rapidly changing external environment (inflation, economic shocks, regulatory change)
• Continuous learning systems updating from potentially shifted incoming data
• Vendor-hosted models (LLM APIs) updated or retired by the provider, often behind an unchanged model name

Consequence types

Type	Example
Financial loss	Underwriting model pricing stale risk relationships
Operational degradation	Claims triage model becoming systematically inaccurate
Customer harm	Fraud model false-positive spike affecting legitimate customers
Regulatory exposure	Model performance no longer meets documented standards

Affected functions

Risk · Actuarial · Credit · Fraud · Claims · Underwriting · Operations

Controls summary

Control	Owner	Effort	Go-live?	Definition of done
Continuous performance monitoring	Technology	Medium	Required	Automated monitoring tracks metrics on 30-day rolling window. Dashboard reviewed monthly. Alerts fire on threshold breach.
Drift detection	Technology	Medium	Required	Statistical drift detection on primary features. Alert threshold defined and documented.
Retraining triggers	Risk	Low	Required	Retraining triggers documented in AI Register. Named owner responsible for acting.
Scheduled periodic review	Risk	Low	Post-launch	AI Register includes review date (max 12 months). Review completed at least once.

---

Layer 3 — Controls detail

A2-001 — Continuous performance monitoring

Owner: Technology | Type: Detective | Effort: Medium | Go-live required: Yes

Track accuracy, precision, recall, F1, or domain-relevant KPIs on a rolling schedule against a held-out validation set. Dashboard reviewed at minimum monthly by the model owner.

A2-002 — Statistical drift detection

Owner: Technology | Type: Detective | Effort: Medium | Go-live required: Yes

Implement PSI (Population Stability Index) on key input features and on the model's score distribution. A common rule of thumb treats PSI above 0.1 as worth investigating and above 0.25 as significant shift; these are conventions, not a standard, so calibrate them per model. A two-sample Kolmogorov–Smirnov test is an alternative for continuous features. Input drift does not prove concept drift: confirming that the relationship the model learned has changed needs ground-truth outcomes, so track performance on labelled outcomes as they arrive (A2-001). Shifts in feature attributions (e.g. SHAP) are a useful early signal where labels arrive late. Alert thresholds documented in the model risk record.

For vendor-hosted LLMs, pin to a dated model version where the provider offers one, track the provider's deprecation schedule, and re-run a fixed evaluation set before any version change and on a schedule.

A2-003 — Retraining triggers

Owner: Risk | Type: Corrective | Effort: Low | Go-live required: Yes

Define thresholds at which performance degradation triggers retraining or revalidation. Document in the model risk record. Include a scheduled revalidation trigger regardless of metric status — quarterly for models driving customer or credit decisions, at least annually otherwise.

Jurisdiction notes: AU — APRA CPS 230 (in force 1 July 2025) — operational risk management for critical operations, including the technology they depend on | EU — AI Act Art. 72 — providers of high-risk systems must run post-market monitoring (Annex III high-risk obligations apply from 2 December 2027 under Regulation (EU) 2026/1744 — see B2) | UK — PRA SS1/23 Model risk management principles for banks (effective 17 May 2024) — expects ongoing monitoring of model performance | US — Federal Reserve SR 11-7 — model risk guidance requires ongoing monitoring, including outcomes analysis

KPIs

Metric	Target	Frequency
PSI on primary features and score	< 0.1 (investigate 0.1–0.25)	Weekly
Model performance vs baseline	< 5% degradation	Monthly

---

Layer 4 — Technical implementation

import numpy as np
from scipy import stats

def calculate_psi(expected, actual, buckets=10, eps=1e-6):
    """Population Stability Index between a baseline and a current sample.

    Bins are fixed from the baseline's quantiles and applied to both samples.
    Rule of thumb: < 0.1 stable, 0.1-0.25 investigate, > 0.25 significant shift.
    """
    expected, actual = np.asarray(expected), np.asarray(actual)
    edges = np.unique(np.quantile(expected, np.linspace(0, 1, buckets + 1)))
    edges[0], edges[-1] = -np.inf, np.inf          # catch values outside the baseline range
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, eps, None), np.clip(a, eps, None)  # avoid log(0) for empty bins
    return float(np.sum((a - e) * np.log(a / e)))

def ks_shift(expected, actual, alpha=0.01):
    """Two-sample Kolmogorov-Smirnov test; True if the distributions differ."""
    return stats.ks_2samp(expected, actual).pvalue < alpha

# Monitoring: Evidently AI, Arize Phoenix, NannyML (performance estimation without labels)
# Feature store: Feast
# Experiment tracking: MLflow, Weights & Biases

---

Incident examples

Models broken by pandemic behaviour (2020): When spending and shopping patterns shifted abruptly in early 2020, models trained on pre-pandemic data — demand forecasting, inventory, fraud and recommendation systems — produced unreliable outputs, and many organisations had to intervene manually or retrain. (MIT Technology Review, "Our weird behavior during the pandemic is messing with AI models", May 2020)

Zillow Offers wind-down (November 2021): Zillow closed its algorithm-driven home-buying business after its pricing models failed to keep up with a shifting housing market, writing down about US$304 million of inventory in Q3 2021 and cutting about a quarter of its workforce. Its CEO cited the unpredictability of forecasting home prices. (Zillow Group Q3 2021 shareholder letter; AP and Bisnow reporting, November 2021)

Same model name, different behaviour (2023): Researchers compared the March and June 2023 versions of GPT-4 behind the same API and found large changes on fixed tasks; on identifying prime versus composite numbers, accuracy fell from 84% to 51%. Teams relying on a hosted model had no visibility of the change. (Chen, Zaharia and Zou, Harvard Data Science Review, 2024)

---

Scenario seed

Context: A bank's fraud detection model has been in production for 18 months. No monitoring framework exists.

Trigger: The risk team receives an unexplained spike in customer complaints about legitimate transactions being declined.

Complicating factor: The model's aggregate accuracy metric (measured quarterly) has not degraded — the drift is concentrated in a specific customer segment not well-represented in the validation set.

Discussion questions: What monitoring would have detected this earlier? How do you investigate drift when aggregate metrics look healthy? Who is accountable for this outcome?

Difficulty: Intermediate | Jurisdictions: AU, Global