AI Governance & Model Risk Management in Practice

AI Governance and Model Risk

AI systems now draft customer replies, summarize contracts, score credit applications, route support tickets, write code, and power agents that take actions. Each brings new kinds of risk: wrong or fabricated answers, unfair outcomes for some groups, leaks of personal data, unsafe actions, regulatory exposure, and silent changes when a vendor updates a model. AI governance is how an organization keeps those risks visible and owned — knowing what AI it uses, for what purpose, with what evidence that it works, under whose responsibility, and with what monitoring.

Good governance is not a committee that blocks launches. It is a lightweight, risk-proportionate process built into product development: higher-risk uses get more evidence and review, lower-risk uses move quickly, and every system has documentation and an owner. Banks have done this for statistical models for decades under model risk management; generative AI extends the same discipline to prompts, retrieval, tools, and vendor models.

TL;DR

Keep an inventory of AI systems with owners, models, data, and purpose.
Document intended use, out-of-scope use, and risk tier for every system.
Require evaluation evidence proportional to risk — quality, safety, robustness, fairness — with release gates and thresholds.
Assign owners, define human oversight, and route high-risk systems through review.
Govern change: model versions, prompts, retrieval data, and tools; re-evaluate before changes reach users.
Monitor in production, handle AI incidents, keep records, and align with frameworks such as the NIST AI RMF, ISO/IEC 42001, and the EU AI Act.

Quick Example

A system record kept in the repository next to the code, reviewed like any other change:

A release gate that blocks deployment when evaluation results miss the documented thresholds:

Core Concepts

Inventory and Ownership

You cannot govern what you cannot see. Maintain an inventory of AI systems — including vendor features embedded in SaaS tools — with the business owner, technical owner, models and versions, data sources, users affected, and risk tier. Every system needs a named owner accountable for its behavior, evidence, and monitoring.

Intended Use and Risk Tiering

Tiers (for example low, medium, high) decide how much evidence, review, oversight, and monitoring a system needs. Document out-of-scope uses explicitly; many incidents come from systems used beyond their tested purpose.

Documentation

Keep documentation in version control near the code, so it changes with the system.

Evaluation Evidence and Release Gates

Evidence should match the risk: task quality and groundedness, safety and policy compliance, robustness to adversarial inputs (red teaming, prompt injection tests), performance across languages and user groups (fairness), privacy leakage, and cost and latency. Encode thresholds as release gates in CI, so regressions block deployment. See LLM Evaluation and AI Guardrails.

Human Oversight

Define where people review, approve, or can override AI outputs, and make sure oversight is real: reviewers need time, information, and authority, and their decisions should be measured (how often they edit or reject outputs). For automated decisions about people, provide explanations and an appeal path.

Third-Party and Foundation Models

Most organizations use models from vendors. Due diligence covers data handling and retention, training on customer data, security, regional processing, model update and deprecation policies, available documentation, and contractual commitments. Pin model versions where possible and re-evaluate before switching. See Third-Party Privacy and Security Reviews.

Change, Monitoring, and Incidents

Change management: model versions, prompts, retrieval corpora, tools, and guardrails are all changes that can alter behavior; run evaluations before release and record approvals. See LLMOps.
Monitoring: quality on sampled production traffic, user feedback and complaints, safety filter triggers, drift in inputs, latency, and cost.
AI incidents: harmful outputs, data leaks, unfair outcomes, or unsafe actions handled like security incidents — triage, containment (feature flags, rollback, disabling tools), communication, and postmortems.
Decommissioning: retire systems deliberately, deleting data and revoking access.

Frameworks and Regulation

Map internal controls to the frameworks relevant to your markets, and involve legal and compliance early. See Responsible Technology and Accessibility Governance.

Best Practices

Make Governance Proportional

Light requirements for low-risk uses, rigorous evidence and review for high-risk ones.

Automate Evidence

Run evaluations in CI, generate documentation from code and configuration, and gate releases on thresholds.

Treat Prompts and Models as Releases

Version, review, evaluate, and roll out changes to prompts, models, and retrieval data like code.

Close the Loop With Production

Monitor real behavior, collect feedback, and feed incidents back into evaluation sets.

Common Mistakes

Governance by Spreadsheet After Launch

Silent Vendor Model Changes

Rubber-Stamp Human Review

FAQ

What is AI governance?

The policies, processes, roles, and tools an organization uses to ensure its AI systems are used for appropriate purposes, perform as intended, comply with laws, and are monitored and accountable throughout their lifecycle.

What is model risk?

The risk of adverse consequences from decisions based on incorrect or misused models, including errors in the model, its data, its implementation, or its application beyond intended use.

What is a model card?

A short document describing a model's intended use, performance across conditions and groups, limitations, and ethical considerations, so users and reviewers understand when it is appropriate.

How does the EU AI Act affect software teams?

It sets risk-based obligations: some uses are prohibited, high-risk systems need risk management, data governance, documentation, human oversight, and conformity assessment, and some systems have transparency duties. Obligations apply in phases, so check which apply to your systems and markets.

How should teams govern generative AI features built on vendor models?

Document intended use and risk, evaluate the full system (prompts, retrieval, tools, model) before release, pin model versions, review vendor terms, monitor production quality, and re-evaluate before changes.