AI Governance for Regulated Environments

TL;DR

• Classify AI risk by consequence, exposure, autonomy, reversibility, and evidence. A customer-visible chatbot and an autonomous credit decision should not face the same controls.
• Governance begins in discovery. The team should know the owner, permitted use, data boundary, evaluation standard, approval path, and recovery plan before production work accelerates.
• A person in the loop only reduces risk when they have enough time, context, authority, and skill to detect and correct a failure.
• Prompt instructions and model-based filters are not security boundaries. Constrain sensitive data, tools, credentials, and consequential actions outside the model.
• Regulation is technology-neutral and changes by jurisdiction. Product teams need current legal, privacy, security, risk, and domain expertise rather than a generic AI compliance checklist.

---

AI governance is the operating system around a consequential product decision. It defines who may approve a use, what evidence is required, which actions are permitted, how the system is monitored, and what happens when it fails.

Treating governance as a launch meeting creates two bad outcomes. Low-risk work waits for unnecessary approval, while high-risk work reaches reviewers after the architecture and customer promise are difficult to change. Proportional governance moves the decision earlier and makes the required evidence explicit.

This chapter is a product framework, not legal advice. Applicable obligations depend on the organisation, data, use case, customer, sector, and jurisdiction.

Classify the use case, not the model

The same model can reformat internal notes or influence a financial decision. Model capability matters, but the use determines the risk.

Assess five dimensions:

Data sensitivity, legal obligations, vulnerability of affected people, and the likelihood of correlated failure cut across all five.

Use a small number of risk classes that match the organisation's existing risk language. A four-level scale can work:

These are starting definitions, not automatic legal categories. A company should map them to its risk appetite and existing control system.

Human review is a control that needs an eval

Adding a reviewer does not automatically lower the risk. Review fails when people lack time, context, authority, or a clear standard. High-volume review can also decay into rubber-stamping.

Test the review system itself:

• Can reviewers detect seeded failures?
• Do they see the evidence needed to make the decision?
• How much time does a careful review take?
• What happens when they disagree with the system?
• Can they stop or reverse the action?
• Does review quality hold at expected volume?

Measure overrides alongside independent outcome quality. A low override rate may mean the system is excellent, or it may mean the control is decorative. The AI product metrics chapter covers this distinction.

Define the governance record

Every production use should have a record that a product, risk, security, legal, or operational reviewer can understand without reconstructing the project from tickets.

Record:

• The intended outcome, users, and prohibited uses
• Accountable outcome and system owners
• Model, configuration, tools, and material dependencies
• Data sources, classifications, locations, retention, and permitted purposes
• Authority boundary and actions requiring approval
• Evaluation set, graders, thresholds, segment results, and known limitations
• Human review design and expected operating load
• Monitoring, alerts, incident path, fallback, and rollback
• Approval decision, conditions, evidence, and next review trigger

This is a use-case register, not merely a model inventory. One model may support several uses with different risks, owners, data, and controls. The Every Agent Needs an Owner chapter provides the operating contract for systems that act.

Put governance gates through the lifecycle

Governance should follow the product decision rather than appear as one final gate.

Discovery gate

Confirm that the problem is worth solving, identify affected people, assign an initial risk class, and involve the specialists required by the use case. A high-consequence idea may need legal, risk, privacy, security, accessibility, or domain input before a prototype touches real data.

Evidence gate

Define the evaluation standard and test the riskiest assumptions. Include failure cases, important segments, misuse, recovery, and the human operating process. A high aggregate score does not excuse failure in a legally or commercially important cohort.

Release gate

Review the actual production boundary: data, permissions, tools, model configuration, monitoring, fallback, incident ownership, and customer communication. Approval belongs to named roles with authority, not a committee whose presence hides accountability.

Change gate

Reassess material changes to models, prompts, tools, data, autonomy, users, jurisdictions, or review processes. Use evals to detect behavioural regressions, but do not treat a passing score as proof that legal or operational risk is unchanged.

Retirement gate

Remove access, credentials, schedules, stored context, and monitoring when the use ends. Preserve the records required for incidents, audit, customer support, or legal obligations. An abandoned agent with live permissions is still a production risk.

Govern data by purpose and flow

"Does this use personal data?" is only the first question. Map the full data flow:

1. Where did the data come from, and for what purpose was it collected?
2. Which transformations, retrieval systems, prompts, and tools expose it to the model?
3. Where is it processed, logged, cached, stored, or reviewed?
4. Who can see the input, intermediate trace, output, and audit record?
5. When is each copy corrected, deleted, or retained?

Publicly accessible information is not automatically free of privacy, copyright, confidentiality, or contractual constraints. De-identification may also be reversible when records are combined with other context. Work with privacy and legal specialists on the actual data and purpose.

Product requirements should state the minimum data needed, allowed purpose, residency and transfer constraints, retention, access controls, and response to data-subject or customer requests. Test those requirements in the complete workflow, including observability tools and human review queues.

Security follows authority

Language models introduce prompt injection and other manipulation risks. The system may interpret instructions hidden in user input, retrieved documents, email, webpages, tool output, or memory. Prompt wording, delimiters, safety policies, and model-based filters can reduce some failures, but they do not enforce authority.

Design as though a model can be manipulated. Limit the consequence of its worst valid action.

Use layered controls:

• Give the system the least data and tool authority needed for the task.
• Preserve the trust level and provenance of content entering context.
• Separate retrieval, decision, and action where the job permits it.
• Validate tool arguments and policy rules outside the model before execution.
• Require approval for high-consequence actions.
• Isolate tenants, credentials, and execution environments.
• Detect repeated, unusual, or unauthorised tool behaviour.
• Make consequential actions idempotent or reversible where possible.
• Test data exfiltration, indirect prompt injection, privilege escalation, and recovery.

Do not let a filtering layer create false confidence. It can remain one signal in a layered design, while identity, least privilege, deterministic policy, sandboxing, and transaction controls enforce the boundary.

The agentic security playbook covers prompt injection, per-task permissions, sandboxing, adaptive attacks, and incident containment. The agent safety inspection turns operational boundaries into circuit breakers, trace audits, drift drills, and performance budgets.

Australian regulated-product starting points

Australian financial and privacy regulation is technology-neutral. Existing obligations continue to apply when AI performs or influences the work.

Current official starting points include:

• APRA's 2026 letter on AI risk, which calls for stronger governance, assurance, operational resilience, oversight, and accountability across APRA-regulated entities.
• ASIC Report 798, which examines the gap between AI adoption and governance among financial services and credit licensees.
• OAIC guidance on privacy and generative AI, which explains how existing Privacy Act obligations apply to personal information used in AI development and training.

These sources do not replace use-case advice. Confirm current obligations with the accountable specialists, particularly where a system affects credit, insurance, superannuation, advice, employment, vulnerable people, or access to essential services.

Measure whether governance works

Track measures that reveal decision quality and operating burden:

• Approval time by risk class and cause of delay
• Conditions attached to approval and whether they close
• Incidents, near misses, and customer harm by use case
• Evaluation regressions and time to restriction or recovery
• Review volume, queue age, reviewer accuracy, and escalation cost
• Unregistered uses, expired approvals, and overdue reviews
• Time required to produce evidence for an audit or investigation

Fast approval is not success if incidents rise. A large register is not success if nobody can identify which systems can act on customer data. Use measures to improve the controls, boundaries, and evidence.

Set review triggers

Calendar reviews are useful, but material change should trigger review sooner. Reassess when:

• The system receives a new tool, data source, user segment, or jurisdiction
• Autonomy or volume increases
• A model or provider changes materially
• Evals or production monitoring reveal a new failure pattern
• A law, regulator expectation, contract, or company risk appetite changes
• Reviewers can no longer keep up with the operating load
• An incident shows that the documented boundary was wrong

The execution and delivery chapter shows how to make these controls part of readiness and release rather than a parallel compliance stream.

The anti-pattern: governance theatre

Governance theatre produces policies, committees, and approval decks without a reliable answer to five basic questions: what is running, who owns it, what it can do, how anyone knows it works, and how it will be stopped.

The opposite failure ships first and reconstructs the risk later. Both separate governance from the product decision.

Good governance leaves an inspectable chain from intended outcome to authority, evidence, approval, operation, and recovery. It gives low-risk work a clear path and forces consequential work to earn trust.