Skip to main content

E3 — Misinformation & Disinformation

Medium severityEU AI Act Art. 50NIST AI 600-1DIGI Disinformation Code (AU)CDSA

Domain: E — Fairness & Social | Jurisdiction: AU, EU, Global

Layer 1 — Start here​

AI systems generate or amplify false, misleading, or deceptive information at scale — creating systemic information integrity risk.

AI dramatically lowers the cost of producing convincing false content at scale. In May 2023, an apparently AI-generated image of an explosion near the Pentagon, spread by a verified account impersonating Bloomberg News, briefly pushed the S&P 500 down about 0.3% before officials confirmed there had been no explosion. EU AI Act Art. 50 sets targeted transparency duties: chatbots must tell people they are dealing with an AI, providers must machine-mark synthetic content, and deployers must disclose deep fakes and AI-generated text published to inform the public.

Do we know which EU AI Act Art. 50 transparency duties apply to each of our AI systems — and do we meet them?

AI can generate convincing false information at scale — fake news, fabricated quotes, synthetic personas. The EU AI Act Art. 50 creates mandatory disclosure obligations. Your organisation faces risk in two directions: AI systems may generate misinformation inadvertently, and synthetic content attributed to your organisation may be used against you.

Layer 2 — Overview​

Likelihood drivers​

  • AI-generated content deployed at scale without human review
  • No content provenance or watermarking on AI-generated outputs
  • No monitoring for AI-generated content misuse or manipulation
  • Synthetic content disclosure obligations not implemented

Consequence types​

TypeExample
Market manipulationFalse AI-generated information eroding shareholder value
Regulatory enforcementEU AI Act synthetic content disclosure obligations
Reputational harmDeepfake or AI-generated false content attributed to organisation
Information integrityAI-generated synthetic personas in public consultations

Affected functions​

Communications · Legal · Compliance · Risk · Marketing

Controls summary​

ControlOwnerEffortGo-live?Definition of done
Synthetic content disclosureComplianceLowRequiredApplicable EU AI Act Art. 50 duties mapped to each touchpoint and met; organisational policy covers disclosure of other AI-generated content. Mechanism implemented and tested before go-live.
Content provenance (C2PA)TechnologyMediumPost-launchC2PA content authentication evaluated. Deployment decision documented. If deployed, official organisational media signed with organisational certificate.
AI content detection for inbound contentComplianceMediumPost-launchAI-generated content detection deployed for high-stakes inbound content. Results reviewed as part of content processing workflow.
Media literacy and staff trainingHRLowPost-launchStaff educated on identifying AI-generated content. Training completed and tracked.

Layer 3 — Controls detail​

E3-001 — Synthetic content disclosure​

Owner: Compliance | Type: Preventive | Effort: Low | Go-live required: Yes

EU AI Act Art. 50 does not require every piece of AI-generated content to be labelled. It requires: (1) providers to design AI systems that interact directly with people so that those people are told they are dealing with an AI, unless this is obvious from the context (Art. 50(1)); (2) providers of AI systems generating synthetic audio, image, video or text to mark outputs in a machine-readable format so they are detectable as AI-generated (Art. 50(2)); and (3) deployers to disclose deep fake image, audio or video content, and AI-generated or manipulated text published to inform the public on matters of public interest — unless the text has undergone human review or editorial control and someone holds editorial responsibility (Art. 50(4)). Art. 50(3) separately covers emotion recognition and biometric categorisation. Many organisations disclose all AI-generated content as a matter of policy, because a person interacting with AI-generated content has a reasonable expectation of knowing it is not human-authored.

Implementation requirements: (1) Scope definition — identify every touchpoint in the product or service where AI-generated content is presented to natural persons: chatbot responses, AI-generated documents, synthetic images, audio narration, video content. Disclosure must be applied consistently — selective disclosure that only covers some touchpoints creates both legal risk and user trust damage; (2) Disclosure mechanism — implement disclosure in a manner that is clear, prominent, and understandable. Acceptable mechanisms: (a) inline label on generated content ("Generated by AI"); (b) system-level disclosure at session start for AI chatbots; (c) metadata embedding for synthetic media (C2PA standard — see E3-002); (d) watermark for synthetic audio and video. Buried disclosures in terms and conditions do not satisfy the EU AI Act requirement; (3) Chatbot identity disclosure — AI chatbots must not deceive users into believing they are human. On direct inquiry ("Are you a human?", "Am I talking to a person?"), the system must disclose it is an AI. This obligation applies even when the chatbot has a persona or human-sounding name; (4) Implementation testing — test the disclosure mechanism before go-live: confirm it appears consistently across all touchpoints, is visible on all device types (mobile, desktop), and cannot be disabled by users in ways that create a misleading impression; (5) Documentation — document the disclosure mechanism, when it was implemented, and how compliance was confirmed. Retain as evidence of EU AI Act Art. 50 compliance.

Jurisdiction notes: AU — no mandatory AI content disclosure obligation as of October 2026 (the Australian Standards for AI announced on 15 July 2026, with legislation expected in 2027, centre on data centres and copyright in the measures announced so far); OAIC has noted AI disclosure as an emerging expectation in privacy policy and consumer communications | EU — EU AI Act Art. 50 — chatbot disclosure (Art. 50(1)), machine-readable marking of synthetic content by providers (Art. 50(2)) and deployer disclosure of deep fakes and AI-generated public-interest text (Art. 50(4)); applies from 2 August 2026 (Art. 113), except that providers of generative systems placed on the market before that date have until 2 December 2026 to meet Art. 50(2) (Art. 111(4), inserted by Regulation (EU) 2026/1744) | US — FTC Act Section 5 — undisclosed AI-generated content in consumer contexts may constitute deceptive practice; no federal mandate as of 2026. Several states considering disclosure requirements

E3-002 — Content provenance (C2PA)​

Owner: Technology | Type: Detective | Effort: Medium | Go-live required: No (post-launch)

C2PA (Coalition for Content Provenance and Authenticity) is an open technical standard for embedding cryptographically signed provenance metadata into content — establishing who created it, when, with what tools, and whether it has been modified. It is the emerging technical infrastructure for AI content attribution and the mechanism for EU AI Act Art. 50 machine-readable disclosure for synthetic media.

Implementation requirements: (1) Standard evaluation — assess whether C2PA is appropriate for the organisation's content types and workflows. C2PA is well-suited to: images, video, audio, and documents. Assess current tool and platform support for C2PA in the content creation and publishing pipeline; (2) Signing infrastructure — if C2PA is deployed, obtain a C2PA certificate from an accredited issuer. All AI-generated official organisational media (press imagery, video content, official publications) should be signed with the organisational certificate. Maintain the private key in a secure key management system; (3) Provenance display — implement a mechanism for readers to verify content provenance (C2PA-compatible viewer, or reference to verification tools). The value of C2PA is only realised if recipients can verify signatures; (4) Inbound content — for organisations that receive external content as part of operations (journalism, research, procurement), C2PA metadata on received content should be checked as part of content processing. Absence of C2PA metadata is not confirmation of inauthenticity, but presence of authentic C2PA metadata is confirmatory; (5) Decision documentation — if C2PA is not deployed, document the assessment and rationale. The decision to defer should reflect a genuine assessment of applicability, not a default non-decision.

Jurisdiction notes: EU — EU AI Act Art. 50(2) — providers of AI systems generating synthetic audio, image, video or text must mark outputs in a machine-readable format so they are detectable as AI-generated (from 2 August 2026; 2 December 2026 for systems placed on the market before that date, Art. 111(4)). C2PA is the emerging implementation standard for this requirement | AU — no mandatory C2PA requirement; voluntary adoption aligns with DISR voluntary AI Safety Standard | US — NIST AI 600-1 — content provenance is identified as a key mitigation for synthetic content risks

E3-003 — AI content detection for inbound content​

Owner: Compliance | Type: Detective | Effort: Medium | Go-live required: No (post-launch)

For organisations where inbound content authenticity is material — news organisations, insurers assessing submitted evidence, courts, financial regulators — AI-generated content detection tools provide a signal for whether inbound content may be synthetic. These tools are probabilistic, not definitive, and must be used as decision-support rather than determinative.

Implementation requirements: (1) Use case scoping — identify which inbound content types are material enough to warrant AI detection screening: submitted documents, photographs as evidence, regulatory submissions, research inputs. Not all inbound content requires screening; prioritise by the consequence of acting on false content; (2) Tooling selection — AI detection tools vary significantly in accuracy and are subject to rapid obsolescence as generation models improve. Select tools with documented false positive and false negative rates against current generation models. Review tool effectiveness annually; (3) Decision-support framing — train staff using AI detection results to understand the probabilistic nature of the output. A high AI-detection score is a flag for investigation — it is not proof the content is fake. A low score is not proof the content is authentic. Human investigation remains required for material content; (4) Investigation protocol — define what happens when AI detection flags content: who investigates, what investigation steps are taken (reverse image search, metadata analysis, source verification), and who makes the final determination. Document decisions on flagged content; (5) Limitations disclosure — where AI detection is used in consequential processes (insurance claims, legal proceedings), disclose its use and its limitations. Relying solely on AI detection in a consequential process without disclosure and without human review is not appropriate.

Jurisdiction notes: AU — Evidence Act obligations — AI detection evidence would require expert testimony on methodology and reliability in legal proceedings | EU — GDPR Art. 22 — if AI detection results in automated decisions affecting individuals, safeguards apply | US — evidence admissibility standards vary by jurisdiction; AI detection findings would require expert evidence foundation

E3-004 — Media literacy and staff training​

Owner: HR | Type: Preventive | Effort: Low | Go-live required: No (post-launch)

Staff who consume, evaluate, and act on information — in finance, in communications, in risk functions — need practical skills to recognise AI-generated content and understand why detection is becoming harder. This is not generic digital literacy training; it must be specific to AI-generated content and current as of the training date.

Implementation requirements: (1) Content — training must cover: (a) how AI-generated text, images, and audio are created and why they are increasingly indistinguishable; (b) specific indicators that may remain: metadata anomalies, C2PA absence, distributional tells in generated images (hands, lighting, backgrounds); (c) detection tool capabilities and limitations — what the tools can and cannot tell you; (d) verification processes — how to check the authenticity of high-stakes content before acting on it; (e) organisational policy on use of AI content detection tools; (2) Targeting by role — not all staff need the same depth. Finance staff need specific training on AI-generated payment instructions and voice cloning. Communications staff need training on synthetic imagery. Research staff need training on AI-generated text. Tailor training to role-specific exposure; (3) Currency — AI content capabilities are advancing at a rapid pace. Training that was current eighteen months ago may be materially outdated. Annual refresh minimum; additional refreshes when significant capability jumps occur (e.g. new voice cloning services become accessible); (4) Practical scenarios — include examples relevant to the organisation's specific risk context. Abstract training about AI deepfakes is less effective than training that uses realistic examples from the industry the staff work in.

Jurisdiction notes: AU — APRA CPS 234 — security awareness training obligation; AI-generated phishing and social engineering is a current threat that training must address | EU — NIS2 — training obligations for entities in scope; AI-generated social engineering is within scope of NIS2 security awareness requirements | US — FFIEC — financial institution security awareness training must reflect current threat landscape

KPIs​

MetricTargetFrequency
AI-generated content presented without disclosureZeroMonitored continuously
C2PA evaluation decision documentedYes — approved or deferred with rationaleAt initial assessment
Staff media literacy training completion> 90% of targeted rolesAnnual
AI detection false positive rate reviewWithin documented acceptable thresholdQuarterly
Chatbot identity disclosure tested100% of AI interfaces tested on direct inquiryAt each release

Layer 4 — Technical implementation​

Synthetic content disclosure — implementation patterns​

from enum import Enum

class ContentType(Enum):
    TEXT = "text"
    IMAGE = "image"
    AUDIO = "audio"
    VIDEO = "video"
    DOCUMENT = "document"

class DisclosureMethod(Enum):
    INLINE_LABEL = "inline_label"      # "Generated by AI" label on content
    SESSION_BANNER = "session_banner"  # Disclosure at session start for chatbots
    METADATA_C2PA = "metadata_c2pa"    # Cryptographic provenance in content metadata
    WATERMARK = "watermark"            # Perceptual watermark for audio/video

DISCLOSURE_POLICY = {
    # content_type -> required_methods
    ContentType.TEXT: [DisclosureMethod.INLINE_LABEL],
    ContentType.IMAGE: [DisclosureMethod.INLINE_LABEL, DisclosureMethod.METADATA_C2PA],
    ContentType.AUDIO: [DisclosureMethod.INLINE_LABEL, DisclosureMethod.WATERMARK],
    ContentType.VIDEO: [DisclosureMethod.INLINE_LABEL, DisclosureMethod.WATERMARK, DisclosureMethod.METADATA_C2PA],
    ContentType.DOCUMENT: [DisclosureMethod.INLINE_LABEL],
}

def apply_disclosure(content: dict, content_type: ContentType) -> dict:
    """
    Apply required disclosure mechanisms to AI-generated content
    before delivery to user.
    """
    required_methods = DISCLOSURE_POLICY[content_type]
    applied = []

    for method in required_methods:
        if method == DisclosureMethod.INLINE_LABEL:
            content["ai_disclosure_label"] = "Generated by AI"
            applied.append(method.value)
        elif method == DisclosureMethod.SESSION_BANNER:
            content["session_disclosure"] = True
            applied.append(method.value)
        elif method == DisclosureMethod.METADATA_C2PA:
            # In production: call C2PA signing service
            content["c2pa_signed"] = False  # placeholder
            applied.append(method.value)
        elif method == DisclosureMethod.WATERMARK:
            content["watermark_applied"] = False  # placeholder — call watermarking service
            applied.append(method.value)

    content["disclosure_applied"] = applied
    content["eu_ai_act_art50_compliant"] = (
        len(applied) == len(required_methods)
    )
    return content

def chatbot_identity_check(user_message: str) -> bool:
    """
    Detect direct identity inquiry — must always disclose AI nature.
    Returns True if message is a direct identity question.
    """
    IDENTITY_INDICATORS = [
        "are you human", "are you a person", "am i talking to",
        "is this a bot", "are you real", "are you ai",
        "are you a robot", "who am i talking to",
    ]
    msg_lower = user_message.lower()
    return any(indicator in msg_lower for indicator in IDENTITY_INDICATORS)

Compliance implementation​

Australia: No mandatory AI content disclosure legislation as of October 2026. On 15 July 2026 the Prime Minister announced mandatory Australian Standards for AI, overseen by a new Office of AI in the Department of the Prime Minister and Cabinet, with legislation expected in 2027; the measures announced so far centre on large AI data centres and copyright rather than content disclosure, so watch the draft legislation. However, the Australian Consumer Law prohibits misleading or deceptive conduct — presenting AI-generated content in a way that creates a false impression of human authorship may constitute a breach. OAIC guidance expects organisations to be transparent about AI use in privacy policies and communications. The voluntary DISR AI Safety Standard includes transparency as a core principle.

EU: EU AI Act Art. 50 is the most prescriptive disclosure requirement globally. Key obligations: (1) providers must design AI systems that interact with people so those people are told they are dealing with an AI, unless obvious from context (Art. 50(1)); (2) providers of systems generating synthetic audio, image, video or text must mark outputs in a machine-readable format (Art. 50(2)); (3) deployers must disclose deep fakes, and AI-generated text published to inform the public on matters of public interest unless it is under human review or editorial control with editorial responsibility (Art. 50(4)). Applies to providers and deployers from 2 August 2026 (Art. 113); providers of systems placed on the market before that date have until 2 December 2026 to meet Art. 50(2) (Art. 111(4), inserted by Regulation (EU) 2026/1744). C2PA is the emerging implementation mechanism for machine-readable marking.

US: No federal AI disclosure mandate as of 2026. FTC guidance on AI and deception (2023) indicates that undisclosed AI personas in consumer contexts may constitute deceptive practice under Section 5. Political advertising: several states require disclosure of AI-generated political content — for example, California AB 2355 requires political committees to label ads generated or substantially altered using AI (from 1 January 2025). California AB 2655, which required large online platforms to remove or label materially deceptive election content, was struck down in August 2025 as preempted by Section 230 of the Communications Decency Act (Kohls v. Bonta, E.D. Cal.); California has appealed the Section 230 ruling to the Ninth Circuit. The FTC has brought enforcement actions against companies for deceptive AI chatbot practices — including cases where chatbots impersonated humans in consumer interactions.

Incident examples​

Fake Pentagon explosion image moves markets (May 2023): On 22 May 2023 an apparently AI-generated image of an explosion near the Pentagon spread on Twitter, amplified by a verified account impersonating Bloomberg News. The S&P 500 fell about 0.3% within minutes and recovered once Arlington County fire and police officials confirmed there had been no explosion — possibly the first time an AI-generated image moved the market. Source: Bloomberg (22 May 2023); NPR (22 May 2023).

EU election deepfakes (2024): AI-generated deepfake videos of politicians making inflammatory statements were distributed during EU elections to manipulate voter perception.

AI-generated celebrity impersonation misinformation (2025): An AI-generated image of former US President Barack Obama DJing in Arizona was shared widely, reaching millions of people and leading many to mistakenly believe the image was real. Demonstrates that AI-generated celebrity impersonation content can spread as misinformation at scale even without malicious coordinated amplification. Source: AIID monitoring run, May 2026.

OpenAI covert influence operation disruptions (2024–2025): OpenAI publicly documented disrupting multiple covert influence operations (IO) that used its models to generate synthetic content for political manipulation — including state-affiliated networks using AI to produce fake social media personas, translated propaganda, and fabricated quotes attributed to real political figures. The documented cases span multiple jurisdictions and demonstrate that AI-generated content is now a routine component of coordinated disinformation campaigns. Source: OpenAI threat intelligence reporting (2024); AIID monitoring run, May 2026.

Coordinated inauthentic behaviour networks using AI (2024–2025): Meta and other platform operators removed multiple coordinated inauthentic behaviour (CIB) networks that used AI to generate synthetic content — including networks linked to conflicts in Ukraine and Russia, and others manufacturing artificial grassroots opposition in policy debates. AI-generated content in CIB operations lowers production costs for influence campaigns and increases volume beyond what human operators alone can sustain. Source: Meta transparency reporting; AIID monitoring run, May 2026.

Scenario seed​

Context: A public consultation on a proposed infrastructure project receives thousands of submissions. The project team notices many submissions use nearly identical language despite purporting to come from different individuals.

Trigger: Investigation reveals the submissions are AI-generated synthetic personas, manufactured to create the appearance of public opposition.

Difficulty: Advanced | Jurisdictions: EU, AU, Global