Skip to main content

B3 — AI Lifecycle Governance Failure

Medium severityISO 42001 Cl. 8NIST AI RMF MANAGE 2EU AI Act Art. 9APRA CPS 230

Domain: B — Governance | Jurisdiction: AU, EU, Global

Layer 1 — Start here​

Inadequate governance across the full AI lifecycle — from development through deployment, monitoring, and decommissioning.

Software development lifecycle disciplines were developed for deterministic software. AI systems are not deterministic — a model update changes behaviour in ways that cannot be fully specified from a change log. Many organisations deploy AI but have no structured process for what happens next: no change management for model updates, no monitoring framework, no formal decommissioning.

Can we demonstrate that every AI system in production has passed a structured pre-deployment testing gate, is actively monitored, and has a defined review and decommissioning process?

Many organisations have deployed AI but have no structured process for what happens after launch. The audit finding means there is no AISDLC framework. What you are approving is implementing a structured lifecycle framework with mandatory gates before deployment and active oversight throughout — analogous to SDLC but adapted for AI.

Layer 2 — Overview​

Risk description​

AI systems keep changing after launch: data shifts, models are retrained, vendors swap the underlying model, and use spreads beyond the original purpose. Governance designed as a one-off approval at go-live misses all of this. Lifecycle governance failure is the absence of gates and ownership at each stage — design, testing, deployment, change, monitoring and retirement — so that a system can drift, change or outlive its approval without anyone deciding that it should.

Likelihood drivers​

  • No AISDLC framework adapted for AI-specific lifecycle
  • Vendor model updates not subject to change management
  • No pre-deployment testing checklist or minimum standards
  • Models run indefinitely without formal reassessment

Consequence types​

TypeExample
Regulatory breachPlacing a high-risk AI system on the market or putting it into service without the required conformity assessment (EU AI Act Art. 43)
Operational harmModel behaviour changes undetected after vendor update
Fairness failureBias not caught without pre-deployment fairness testing

Affected functions​

Technology · Risk · Compliance · Internal Audit · Procurement

Controls summary​

ControlOwnerEffortGo-live?Definition of done
AISDLC frameworkTechnologyHighPost-launchDocumented AISDLC framework covering all lifecycle phases approved by Risk and actively used for new deployments.
Pre-deployment testing checklistTechnologyLowRequiredStandardised checklist (accuracy, fairness, robustness, security, explainability) completed and signed off for every AI system before go-live.
Vendor model change notificationProcurementLowPost-launchAll AI vendor contracts include advance notice requirement for material model changes. Minimum 30 days. Present in all current contracts.
Decommissioning protocolTechnologyLowPost-launchFormal decommissioning process including data deletion, documentation archival, and AI Register update exists and is followed.

Layer 3 — Controls detail​

B3-001 — Pre-deployment testing checklist​

Owner: Technology | Type: Preventive | Effort: Low | Go-live required: Yes

Require documented completion of a standardised testing checklist before any AI system goes into production. The checklist enforces minimum quality and safety standards across every deployment — not through judgment, but through gates that must be passed. A checklist item without a defined standard is not a control; each item must have a specific, verifiable definition of done.

Minimum checklist items: (1) Accuracy — performance metrics on held-out test set documented and meet defined thresholds; (2) Fairness — disaggregated performance metrics across demographic subgroups documented, no material unexplained disparity; (3) Robustness — tested against out-of-distribution inputs and adversarial examples; (4) Explainability — outputs can be explained to the level required for the use case (adverse action notices, regulatory explanation); (5) Security — adversarial input testing completed for models with external-facing interfaces; (6) Data governance — training data provenance documented, data processing agreements confirmed; (7) Compliance sign-off — written sign-off from Legal confirming regulatory obligations assessed; (8) Monitoring configured — production monitoring dashboard active before go-live, not after.

Note: Tiffany Treloar (Change Management, April 2026) specifically recommended adding entry and exit criteria for AI pilots as an explicit checklist item — the pilot phase is where real stress-testing happens and go/no-go criteria before production should be formally named. This should be a mandatory section for any system that ran as a pilot before production deployment.

Jurisdiction notes: AU — APRA CPS 230 does not mention AI, but its operational risk and change management requirements apply to AI systems like any other technology supporting critical operations; APRA's letter to industry on AI (30 April 2026) sets expectations on governance, operational resilience and assurance (see B2). Fairness and accuracy testing supports Privacy Act and anti-discrimination law obligations | EU — EU AI Act Art. 9 — the risk management system for high-risk AI must include testing before the system is placed on the market or put into service. Art. 15 — accuracy, robustness, and cybersecurity measures must be demonstrated | Global — NIST AI RMF MEASURE 2.3 — performance and assurance criteria are measured and demonstrated for conditions similar to deployment; MANAGE 1.1 — a documented determination of whether deployment should proceed

B3-002 — AISDLC framework​

Owner: Technology | Type: Preventive | Effort: High | Go-live required: No (post-launch)

Implement a structured AI System Development Lifecycle (AISDLC) framework that covers the full lifecycle: problem definition, data acquisition and governance, model development and testing, deployment, monitoring, and decommissioning. The AISDLC is the organisational infrastructure that makes the pre-deployment checklist (B3-001) sustainable at scale.

Lifecycle phases and minimum governance requirements: (1) Problem definition — use case documented, risk rated, legal and compliance consulted; (2) Data — provenance documented, PIA completed if personal data involved, data quality assessed; (3) Development — model trained, evaluated, and version-controlled; (4) Testing — full pre-deployment checklist (B3-001) completed; (5) Deployment — AI Register entry created, RACI documented, monitoring configured; (6) Operations — performance and fairness metrics monitored, model change log maintained; (7) Review — formal reassessment at defined intervals or on material drift; (8) Decommissioning — formal shutdown including data deletion, documentation archival, AI Register update.

The AISDLC is a post-launch control because building the framework takes longer than implementing the checklist. The checklist should be implemented immediately; the AISDLC formalises and extends it.

Jurisdiction notes: AU — APRA CPS 230 — operational risk management, including change management, applies across the life of the technology supporting critical operations, AI included | EU — EU AI Act Art. 9 — risk management system must be continuous throughout the lifecycle. Art. 72 — post-market monitoring is mandatory for high-risk AI | Global — ISO 42001 Cl. 8 — AI system lifecycle management is a core requirement of the AI management system standard

B3-003 — Vendor model change notification​

Owner: Procurement | Type: Preventive | Effort: Low | Go-live required: No (post-launch)

Require all AI vendor contracts to include advance notification of material model changes — typically a minimum of 30 days. A vendor that can update the underlying model without your knowledge or testing undermines your ability to govern the AI system you have deployed.

What constitutes a material change: (1) replacement of the underlying foundation model; (2) significant changes to model weights through retraining; (3) changes to output format or scoring methodology; (4) changes to data inputs used at inference; (5) changes to the vendor's AI sub-processors. The contract clause should define these categories explicitly rather than leaving "material" undefined.

Implementation: include in all new AI vendor contracts; audit existing contracts and negotiate amendments for material legacy vendors; treat notification as a trigger for change management — a vendor model update should go through the same change assessment as an internal model update.

Jurisdiction notes: AU — APRA CPS 230 — entities must manage the risks of material service providers through formal agreements and ongoing monitoring; CPS 230 does not itself oblige a vendor to give notice of model changes, so the obligation has to be written into the contract | EU — EU AI Act Art. 26(5) — deployers of high-risk systems must monitor operation against the instructions for use and inform the provider of risks; Art. 25 — a deployer that substantially modifies a high-risk system can itself become its provider | Global — NIST AI RMF GOVERN 6.1 — policies addressing third-party AI risk; change notification is a practical way to meet it

B3-004 — Decommissioning protocol​

Owner: Technology | Type: Preventive | Effort: Low | Go-live required: No (post-launch)

Implement a formal process for retiring AI systems from production. Deprecated AI models running beyond their intended lifecycle produce increasingly unreliable outputs with no monitoring and no accountable owner — creating silent operational risk.

Decommissioning checklist: (1) decommission decision documented and approved by accountable owner; (2) all data processed by the system handled according to retention policy — personal data deleted or anonymised per Privacy Act / GDPR obligations; (3) system removed from production environment (not just disabled — confirm removal); (4) AI Register entry updated with decommission date and status; (5) documentation archived — model card, testing records, incident log, retained for regulatory minimum period; (6) vendor contract terminated or amended; (7) monitoring and alerting configurations removed to prevent false alarms.

Jurisdiction notes: AU — Privacy Act 1988 APP 11 — personal information no longer needed must be destroyed or de-identified. Retaining training data or inference logs beyond the required period creates ongoing compliance exposure | EU — GDPR Art. 5(1)(e) — storage limitation principle; data must not be kept longer than necessary. EU AI Act Art. 12 — high-risk systems must log events automatically; Art. 19 (providers) and Art. 26(6) (deployers) require those logs to be kept for at least six months, longer where other law requires — reconcile this with deletion duties before decommissioning | US — CCPA right to deletion — AI systems processing California consumer data may receive deletion requests that require system-level response

KPIs​

MetricTargetFrequency
New AI deployments with completed pre-deployment checklist100%Per deployment
AI pilot deployments with documented entry/exit criteria100%Per pilot
AI vendor contracts with change notification clause100% of new contractsQuarterly audit
AI systems decommissioned via formal protocol100% — no silent retirementsContinuous
AISDLC framework adoption rate for new deployments100% once framework publishedAfter framework launch

Layer 4 — Technical implementation​

Pre-deployment testing checklist — structured template​

from dataclasses import dataclass, field
from typing import Literal

CheckStatus = Literal["pass", "fail", "not-applicable", "pending"]

@dataclass
class PreDeploymentCheckItem:
    category: str
    check: str
    status: CheckStatus
    evidence: str           # Location of supporting evidence (doc, test report, etc.)
    reviewer: str           # Name and role of person who verified
    date_completed: str     # ISO 8601
    notes: str = ""

@dataclass
class PreDeploymentChecklist:
    system_id: str
    system_name: str
    deployment_target_date: str
    checklist_owner: str    # Named person accountable for checklist completion

    checks: list[PreDeploymentCheckItem] = field(default_factory=list)
    compliance_signoff_obtained: bool = False
    compliance_signoff_by: str = ""
    compliance_signoff_date: str = ""
    overall_status: CheckStatus = "pending"

    def is_go_live_approved(self) -> bool:
        if not self.checks:
            return False                     # an empty checklist is not a passed one
        covered = {c.category for c in self.checks}
        missing = {cat for cat, _ in STANDARD_CHECKS if cat != "Pilot criteria"} - covered
        outstanding = [c for c in self.checks if c.status not in ("pass", "not-applicable")]
        return not missing and not outstanding and self.compliance_signoff_obtained

# Example minimum checklist items
STANDARD_CHECKS = [
    ("Accuracy",        "Performance metrics on held-out test set meet defined thresholds"),
    ("Fairness",        "Disaggregated metrics documented across all relevant demographic subgroups"),
    ("Fairness",        "No material unexplained disparity without accepted residual risk decision"),
    ("Robustness",      "Tested against out-of-distribution inputs"),
    ("Explainability",  "Output explanation mechanism confirmed for use case requirements"),
    ("Security",        "Adversarial input testing completed for external-facing interfaces"),
    ("Data",            "Training data provenance documented"),
    ("Data",            "Data Processing Agreement confirmed for all data sources"),
    ("Privacy",         "PIA/DPIA completed if personal information processed"),
    ("Governance",      "AI Register entry created with named owner"),
    ("Governance",      "RACI documented and accountable owner acknowledged"),
    ("Monitoring",      "Production monitoring dashboard configured and active"),
    ("Pilot criteria",  "Entry and exit criteria for pilot phase documented (if applicable)"),
    ("Pilot criteria",  "Go/no-go decision for production documented with approver (if applicable)"),
]

Vendor model change management — notification handler​

from dataclasses import dataclass

@dataclass
class VendorModelChangeNotification:
    vendor: str
    system_id: str                # AI Register reference
    change_type: str              # "foundation_model_replacement" | "weight_update" | "output_format" etc.
    change_description: str
    planned_change_date: str      # ISO 8601
    notification_received_date: str
    days_notice: int

    def requires_change_assessment(self) -> bool:
        HIGH_IMPACT_TYPES = {"foundation_model_replacement", "weight_update", "scoring_methodology"}
        return self.change_type in HIGH_IMPACT_TYPES

    def is_compliant_notice(self, minimum_days: int = 30) -> bool:
        return self.days_notice >= minimum_days

def handle_vendor_change_notification(notification: VendorModelChangeNotification) -> dict:
    actions = []

    if not notification.is_compliant_notice():
        actions.append(f"BREACH: Vendor provided {notification.days_notice} days notice — minimum is 30. Escalate to Procurement and Legal.")

    if notification.requires_change_assessment():
        actions.append("Trigger change assessment — assign Technical Owner to evaluate impact")
        actions.append("Schedule regression testing before change date")
        actions.append("Brief Accountable Owner")

    actions.append("Update AI Register change log")

    return {
        "system_id": notification.system_id,
        "actions": actions,
        "change_assessment_required": notification.requires_change_assessment(),
        "go_live_hold": notification.requires_change_assessment(),
    }

Compliance implementation​

Australia: APRA CPS 230 (in force 1 July 2025) — operational risk management, including change management and the management of material service providers, applies to AI systems supporting critical operations. APRA's letter to industry on AI (30 April 2026) sets expectations for boards and accountable executives on AI governance, operational resilience, supplier concentration and assurance. ASIC REP 798 Beware the gap (October 2024) reviewed 624 AI use cases at 23 licensees and found AI adoption outpacing governance and risk management.

EU: EU AI Act Art. 9 — risk management system must be continuous and iterative across the full lifecycle of a high-risk AI system. Art. 72 — post-market monitoring plan is mandatory and must be proportionate to the risk. Art. 12 — logging requirements ensure auditability throughout the operational lifetime. ISO 42001 Cl. 8 — the AI management system standard specifies lifecycle management requirements including planning, implementation, monitoring, and continual improvement.

US: Federal Reserve SR 11-7 / OCC Bulletin 2011-12 (model risk management) — model governance applies throughout the life of a model: validation before use, ongoing monitoring, and an inventory that tracks models through retirement. NIST AI RMF MANAGE function — practices for responding to, monitoring and decommissioning AI systems (MANAGE 2.4 covers superseding, disengaging or deactivating systems).

Tools and references: ISO 42001 · NIST AI RMF · APRA CPS 230 · MLflow (model lifecycle tracking) · DVC (data version control) · Weights & Biases (experiment tracking and model registry)

Incident examples​

Vendor model update rolled back after release (April 2025): OpenAI pushed an update to GPT-4o in ChatGPT that made the model markedly sycophantic — validating doubts, fuelling anger and encouraging impulsive decisions. It began rolling the update back within days and published two postmortems, acknowledging that its pre-release checks had not caught the problem and that it had over-weighted short-term user feedback. Every organisation building on the model inherited the change without notice; only those with their own regression tests would have seen it before their users did. (OpenAI, "Sycophancy in GPT-4o" and "Expanding on what we missed with sycophancy", April–May 2025)

Retired code that was never removed (Knight Capital, August 2012): Knight Capital stopped using an automated order-routing function called Power Peg in 2003 but never deleted it. In 2012 a new release reused the flag that had activated it, and one of eight servers had not received the new code; the dormant logic woke up and sent more than 4 million executions in 154 stocks within about 45 minutes, causing losses that forced the firm's rescue. The SEC fined Knight US$12 million under the market access rule. The system was not machine learning, but the lesson is the decommissioning one: retire a model by removing it, not by switching it off. (SEC order and press release 2013-222, 16 October 2013)

Scenario seed​

Context: A vendor silently upgrades the underlying LLM in a deployed AI product from one model version to another without notification.

Trigger: Customer service staff notice the AI's responses have changed in tone and accuracy. The product owner investigates and discovers the change.

Complicating factor: The vendor contract has no change-notification clause, and the organisation kept no baseline evaluation results, so it cannot show how much behaviour has changed or since when.

Discussion questions: What should the contract have required? What evidence would let you detect and size the change yourself? Who decides whether to keep using the product while the change is assessed? Which customers may have been affected, and do any of them need to be told?

Difficulty: Foundational | Jurisdictions: AU, EU, Global