How to build an AI governance framework that holds up at runtime

An AI governance framework is the set of policies, roles, inventories, risk tiers, controls, monitoring and incident processes an organisation uses to decide which AI systems it builds or uses and to show that it runs them safely and lawfully. In 2026 most enterprises assemble one from three reference points: the NIST AI Risk Management Framework, the ISO/IEC 42001 management system standard and the obligations of the EU AI Act.

The reference points are not the hard part. They overlap more than they differ, and published crosswalks connect them. The hard part is that a governance framework written in 2023 or 2024 was designed for chatbots and predictive models reviewed before launch, while the AI that staff use in 2026 includes agents that call tools, spend money and change records in real time. A policy that says “only approved models and tools” is now a runtime question, decided per request, and most frameworks have nothing in the request path to answer it.

This explainer is for the people who have to build that framework: platform, security, risk and compliance leads at organisations that deploy generative AI. It covers what each reference framework actually requires, how they map onto one control set, the seven components every framework needs, what changes for agentic AI governance, and how written policy becomes enforced control. GenAI Brief’s earlier pieces on the EU AI Act’s model rules and shadow AI cover two of the pressures behind it in more depth.

What an AI governance framework is, and what it is not

An AI governance framework answers four questions continuously: what AI do we have, how risky is each use, who is accountable for it, and how do we know the rules are being followed. Everything else, from committees to model cards, exists to answer one of those four.

It is narrower than “responsible AI”, which also covers values and product design choices, and broader than “AI security”, which covers attacks and data loss but not, for example, whether a credit model’s use is lawful. It is also distinct from model governance in the machine learning operations sense (versioning, approvals, rollback). Model governance is one component of the framework, not the framework.

Three properties separate a working framework from a policy document:

• It is complete. It covers every AI system in use, including third-party tools and AI features inside existing SaaS, not only the models the organisation trains.
• It is proportionate. Controls scale with risk, so a summarisation helper and a hiring screen do not go through the same review.
• It is enforced and evidenced. Each policy statement maps to a control that operates without relying on people remembering it, and that control leaves records an auditor or regulator can inspect.

The third property is where most frameworks fail. The NIST AI RMF calls GOVERN “a cross-cutting function that is infused throughout AI risk management”, which is right, but a cross-cutting function still needs a point of enforcement. For generative AI that point is usually the path a request takes to a model.

The reference frameworks: NIST AI RMF, ISO/IEC 42001, the EU AI Act and the OECD principles

Four documents shape almost every enterprise AI governance framework. They do different jobs, and it helps to be precise about which job each does before mapping them.

NIST AI RMF 1.0 and the Generative AI Profile

NIST released AI RMF 1.0 as NIST AI 100-1 on 26 January 2023. The document describes itself as “voluntary, rights-preserving, non-sector-specific, and use-case agnostic”. Its core is four functions: GOVERN (culture, policy, accountability), MAP (context and intended use), MEASURE (testing and metrics) and MANAGE (prioritising, responding and recovering). Those functions break into 19 categories and 72 subcategories, and each subcategory is an outcome rather than a control. GOVERN 1.6, for example, reads: “Mechanisms are in place to inventory AI systems and are resourced according to organizational risk priorities.”

The framework also defines the characteristics of trustworthy AI that most governance policies quote: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy enhanced, and fair with harmful biases managed.

The Generative AI Profile, NIST AI 600-1, followed in July 2024. It names twelve risks that are unique to or made worse by generative AI: CBRN information or capabilities; confabulation; dangerous, violent or hateful content; data privacy; environmental impacts; harmful bias or homogenisation; human-AI configuration; information integrity; information security; intellectual property; obscene, degrading or abusive content; and value chain and component integration. It then attaches more than 150 suggested actions to the AI RMF subcategories, focused on four themes its working group prioritised: governance, content provenance, pre-deployment testing and incident disclosure.

The profile’s actions are the most practical part of the NIST material for a framework builder. GV-1.6-001 asks organisations to “enumerate organizational GAI systems for incorporation into AI system inventory”. GV-1.3-001 lists factors for defining risk tiers. MG-2.4-004 asks for criteria that warrant deactivating a generative AI system. MG-4.3-002 asks for procedures to “record and track GAI system reported errors, near-misses, and negative impacts”.

NIST’s programme page states, as of October 2026, that “the AI RMF 1.0 is being revised as part of the White House AI Action Plan”, with no published timeline, and that a concept note for a profile on trustworthy AI in critical infrastructure was released on 7 April 2026. A framework built on AI RMF 1.0 today should expect to be remapped once, not rebuilt.

ISO/IEC 42001 and its companions

ISO/IEC 42001:2023, published on 18 December 2023 by the joint ISO/IEC committee on AI (JTC 1/SC 42), “specifies the requirements and provides guidance for establishing, implementing, maintaining and continually improving an AI management system”. It applies to any organisation “that provides or uses products or services that utilize AI systems”, which covers almost every enterprise.

Its structure will be familiar to anyone who has run ISO/IEC 27001. Clauses 4 to 10 cover context, leadership, planning, support, operation, performance evaluation and improvement. Annex A lists 38 reference controls under nine objectives: policies related to AI (A.2), internal organisation (A.3), resources for AI systems (A.4), assessing impacts of AI systems (A.5), the AI system life cycle (A.6), data for AI systems (A.7), information for interested parties (A.8), use of AI systems (A.9) and third-party and customer relationships (A.10). Organisations select controls through a statement of applicability, as they would for information security.

The difference from NIST is certification. An accredited body can audit an organisation against ISO/IEC 42001 and issue a certificate, which is why the standard increasingly appears in enterprise procurement questionnaires. Two companions matter. ISO/IEC 42005:2025, published in May 2025, gives guidance on AI system impact assessments, the process behind Annex A.5. ISO/IEC 42006:2025, published in July 2025, sets requirements for the bodies that audit and certify against 42001.

The EU AI Act after the Digital Omnibus

The EU AI Act is the only one of the four that is law. It assigns duties by role (provider, deployer, importer, distributor) and by risk (prohibited practices, high-risk systems, transparency obligations, general-purpose models). The Digital Omnibus on AI, Regulation (EU) 2026/1744, entered into force on 27 July 2026 and moved several dates. As summarised by the Future of Privacy Forum, the position as of October 2026 is:

The Omnibus also softened Article 4. Providers and deployers now “shall take measures to support the development of AI literacy”, and the text clarifies that this does not require guaranteeing any particular level for any individual. The duty still applies to every deployer today.

For framework builders, the high-risk chapter is the template even before it applies. Providers need a risk management system (Article 9), automatic record-keeping (Article 12), human oversight by design (Article 14) and a quality management system (Article 17), plus post-market monitoring (Article 72) and serious incident reporting (Article 73). Deployers must use systems according to instructions, assign oversight to people with “the necessary competence, training and authority”, monitor operation and keep automatically generated logs under their control “for a period appropriate to the intended purpose” of at least six months (Article 26). Some deployers, including public bodies and those using AI for credit scoring or life and health insurance pricing, must also run a fundamental rights impact assessment (Article 27).

The European AI Office, inside the Commission, enforces the general-purpose model rules directly and prepares guidelines and codes of practice; national market surveillance authorities handle most AI systems. GenAI Brief’s one-year review of the model obligations covers the provider side in detail.

OECD principles and Singapore’s agentic framework

The OECD AI Principles, adopted in 2019 and updated in May 2024, have 47 adherents. Their five values-based principles (inclusive growth and well-being; human rights and democratic values, including fairness and privacy; transparency and explainability; robustness, security and safety; accountability) sit behind the trustworthiness language in both NIST and the EU AI Act. They do not create controls, but they are the right source for the principles section of a corporate AI policy, because regulators across jurisdictions recognise them.

The newest reference is the most relevant to 2026. Singapore’s Infocomm Media Development Authority launched a Model AI Governance Framework for Agentic AI at Davos on 22 January 2026. It organises guidance across four dimensions: assessing and bounding risks upfront, including “placing limits on agents’ powers such as agents’ autonomy and access to tools and data”; making humans meaningfully accountable “by defining significant checkpoints at which human approval is required”; implementing technical controls throughout the agent life cycle, “such as baseline testing and controlling access to whitelisted services”; and enabling end-user responsibility through transparency and training. It is voluntary, and it is the first government document written specifically for agentic AI governance.

How NIST AI RMF, ISO 42001 and the EU AI Act map to each other

The useful way to combine the frameworks is to stop treating them as three checklists and treat them as three views of one control set. Build the control set around the components the organisation actually operates, then record which NIST subcategory, ISO clause or control, and EU AI Act article each component satisfies. One control, three citations.

NIST maintains a list of crosswalks that do part of this work, including AI RMF to ISO/IEC 42005 and to ISO/IEC 23894 (both from the US standards body INCITS, August 2025), NIST AI 600-1 to Singapore’s AI Verify (May 2025), and a 2023 crosswalk relating the AI RMF’s trustworthiness characteristics to the OECD Recommendation and the EU AI Act proposal. The table below is a condensed component-level mapping drawn from the primary texts.

Figure 1: The four reference points do different jobs; a single control set can cite all of them.

Two practical consequences follow. First, an organisation with ISO/IEC 27001 already has most of the management-system machinery (document control, internal audit, management review, corrective action) that ISO/IEC 42001 needs, so the incremental work is the AI-specific Annex A controls and the impact assessment process. Second, the EU AI Act’s high-risk articles are the strictest expression of controls the other two describe loosely. Designing the inventory, logging and oversight components to the Article 12 and Article 26 standard means the framework is already shaped for 2027 when the Annex III obligations arrive.

The seven components of an enterprise AI governance framework

Every mature enterprise AI governance framework has the same seven components, whatever it calls them. The order matters: each depends on the one before.

1. An AI inventory that includes what nobody registered

The inventory is the foundation, and the most commonly faked component. A spreadsheet of projects that teams volunteered is not an inventory. NIST AI 600-1 asks inventory entries to include data provenance, known issues, human oversight roles, rights considerations and “underlying foundation models, versions of underlying models, and access modes” (GV-1.6-003). It also asks organisations to define exemptions explicitly for generative AI embedded in application software (GV-1.6-002), which is an admission that AI features inside SaaS are part of the inventory problem.

A complete inventory combines three sources: declared systems (registration at project start), observed traffic (which models and providers requests actually reach, and from whom) and discovered endpoints (AI apps and MCP servers installed on company machines). The second and third sources need tooling; the shadow AI analysis explains why network logs alone miss desktop apps and local tool servers.

2. Risk tiers that decide the depth of review

Risk tiers turn one framework into proportionate treatment. Three or four tiers are usually enough: prohibited (EU Article 5 practices and anything the organisation chooses to ban), high (EU Annex III categories, decisions about individuals, agents with write access to production systems), limited (customer-facing generative AI with transparency duties) and minimal (internal productivity uses with no personal data). NIST’s GV-1.3-001 lists factors to weigh, including “dependencies between GAI and other IT or data systems”, harm to fundamental rights and the possibility of malicious use.

Each tier should fix three things: which impact assessment applies (none, a short form, or a full ISO/IEC 42005-style assessment), who approves, and which runtime controls are mandatory. The last item is the one most frameworks omit.

3. Policies short enough to be enforced

A typical policy set has an AI principles statement (where the OECD language fits), an acceptable-use policy for staff, a development and procurement standard, and a data policy for AI. The test of each clause is whether a control can enforce it. “Employees must not paste confidential data into unapproved AI tools” is enforceable only if approved tools are distinguishable from unapproved ones at the point of use, and if confidential data can be detected in transit. Policies that cannot pass that test should be rewritten or explicitly marked as training-only.

4. Roles with named owners

The EU AI Act and ISO/IEC 42001 both assume named accountability. A workable split is an executive owner (often the chief risk or chief data officer), a cross-functional AI governance council that sets policy and approves high-risk uses, a system owner for each inventory entry who answers for its risk assessment and monitoring, and a platform owner who runs the enforcement layer. Article 26(2) adds a specific requirement for high-risk deployments: the people assigned to human oversight must have the competence, training and authority to act, which means oversight cannot be a nominal sign-off by someone with no power to stop the system.

5. Controls, split into review-time and run-time

Review-time controls happen before launch: impact assessments, evaluations, red-teaming, supplier due diligence, approval gates. Run-time controls operate on every request: which identities may call which models, which tools an agent may use, how much a team may spend, what content is blocked or redacted, and what is logged. Most written frameworks are rich in review-time controls and thin on run-time ones, because run-time controls require infrastructure. For generative AI the risk is concentrated at run time, where prompts carry data and agents take actions.

6. Monitoring that produces evidence

Monitoring has two outputs: operational signals (cost, error rates, guardrail triggers, unusual usage) and evidence (records that show controls operated). ISO/IEC 42001 control A.6.2.8 covers recording event logs; EU AI Act Article 12 requires high-risk systems to log events automatically, and Article 26(6) requires deployers to retain those logs for at least six months. The design point is to log policy decisions as well as requests: which key, which model, which guardrail fired, which tool call was refused. Separate administrative audit trails, recording who changed a policy, matter as much as request logs.

7. Incident response that knows what an AI incident is

AI incidents include confabulated output that caused harm, data exposure through prompts, an agent taking an unintended action, a jailbreak, a supplier model change that broke behaviour, and cost runaways. NIST AI 600-1 asks for escalation procedures when deactivation criteria are met (MG-2.4-002), after-action reviews (MG-4.3-001) and legally compliant incident reporting (MG-4.3-003). EU AI Act Article 73 requires providers of high-risk systems to report serious incidents to market surveillance authorities, generally within 15 days of becoming aware. The minimum technical capability behind all of this is a kill switch: the ability to revoke a key, disable a model or block a tool for one team or the whole organisation in minutes, without a redeploy.

Agentic AI governance: what changes when systems act

Agentic AI governance is the part of the framework that most 2024-era documents lack. An agent is a system that plans, calls tools and acts on the results, often with credentials that let it change data, send messages or spend money. Three properties make agents different from the systems the original frameworks had in mind.

• The decision happens at run time. A model reviewed before launch behaves within a range; an agent composes tool calls dynamically, so the set of actions it takes cannot be fully enumerated in an approval meeting.
• Identity is delegated. An agent acts on behalf of a user or service, frequently through an MCP server holding a token. The question “who did this?” has two answers, and logs need both.
• Tools are the new attack surface. The OWASP Top 10 for Agentic Applications, published on 9 December 2025, centres on risks such as agent goal hijacking through injected content, misuse of legitimate tools, and identity and privilege abuse. Each of these is a run-time failure that a pre-launch review cannot prevent.

IMDA’s agentic framework answers with “bounding” as the first dimension: choose use cases where errors are recoverable, and limit autonomy and access to tools and data. That translates into four concrete controls for any agent in the inventory: a scoped identity per agent or per team; an allow-list of tools, deny-by-default; spending and rate ceilings per identity; and defined checkpoints, by action type, where a human approves before execution.

Figure 2: Each step up in autonomy adds controls that only operate at run time.

Agents also change the inventory problem. Coding agents and desktop assistants now arrive on employee laptops with MCP servers configured locally, which is why GenAI Brief’s explainer on MCP gateways argues that tool access needs the same central control point as model access. A framework that inventories models but not tools has inventoried half of the agent.

From paper policy to runtime control

The gap between paper policy and runtime enforcement is the defining weakness of enterprise AI governance in 2026. Closing it means translating each enforceable policy clause into a control that sits in the path every AI request takes, and making that control produce evidence by default.

The translation is mechanical once written down:

Figure 3: A policy clause becomes governance only when a control in the request path enforces it and leaves a record.

Why the gateway is the natural enforcement point

An AI gateway sits between applications and model providers, authenticates callers, applies policy and records each request. GenAI Brief’s explainer on what an AI gateway is covers the architecture. For governance, its value is that every row of the table above can be enforced in one place, once, rather than reimplemented in every application.

Several products occupy this layer. Microsoft’s AI gateway in Azure API Management offers an llm-token-limit policy for per-consumer token limits and quotas, content safety checks through Azure AI Content Safety, and prompt and completion logging to Azure Monitor, and can govern MCP servers. The LiteLLM proxy supports budgets and rate limits on keys, users and teams, with options to fail closed when spend cannot be verified. Kong and Cloudflare offer AI gateway features on top of their API platforms. The right choice depends on where workloads run and how much of the control set the gateway covers natively.

Worked example: Bifrost as the enforcement layer

Bifrost, an open-source AI gateway written in Go by Maxim AI, is a useful worked example because its documentation covers most rows of the translation table directly, and because it can be self-hosted, so prompts and logs stay inside the organisation’s network. Its AI governance capabilities map closely onto the framework components above.

The identity layer is the virtual key, which the docs call “the primary governance entity”. Each key carries model and provider filtering, its own budgets, token and request rate limits, an optional expiry and an active or inactive switch, and belongs to one team or customer (in Enterprise, also a business unit or user). Turning a key off is the kill switch from component 7. Budgets are hierarchical: a request must pass every applicable budget, so a team ceiling holds even if an individual key has headroom.

For agents, MCP tool filtering attaches a tool allow-list to each virtual key and is deny-by-default: a key with no MCP configuration gets no tools except from clients explicitly marked as allowed by default. That is IMDA’s “controlling access to whitelisted services” expressed as configuration.

The content layer is guardrails, which validate inputs and outputs for model traffic and MCP tool executions. The documented options include Gitleaks-backed secrets detection, custom regex with a PII template, LLM-as-judge prompt guardrails for organisation-specific policies, and integrations such as Microsoft Presidio, AWS Bedrock Guardrails, Azure Content Safety and Google Model Armor. Built-in observability records each request with inputs, outputs, tokens, cost and latency, and stores the redacted form when guardrail redaction is on. Separately, audit logs record administrative activity (who changed what, and when), can be signed with an HMAC key, and can be archived to S3 or GCS for long-term retention. That split between request logs and admin audit trails is the evidence pair that ISO/IEC 42001 and Article 26 reviews ask for.

At scale, Enterprise access profiles package providers, models, budgets, rate limits and MCP tools into a reusable policy attached to roles, teams or business units, so a policy change propagates without issuing new keys. Performance matters for a control that sits on every request: Bifrost’s published benchmark reports 11 µs of added overhead per request at 5,000 requests per second on an AWS t3.xlarge. Guardrails, audit logs, access profiles and role-based administration are part of the Enterprise tier.

Reaching the laptop

A gateway governs only traffic configured to reach it. Desktop chat apps, browser AI and coding agents running on personal keys bypass it, which leaves the inventory incomplete and the acceptable-use policy unenforced. The response is to move the routing decision onto the machine.

Bifrost Edge applies that model as the endpoint layer of the same gateway. According to its documentation, it runs on macOS, Windows and Linux and routes AI traffic from desktop chat apps, browser AI, coding agents and their MCP servers through the organisation’s Bifrost, so the virtual keys, budgets, rate limits, guardrails and audit logs already configured at the gateway apply on the laptop. App governance lets administrators allow or block AI apps fleet-wide, and MCP governance builds an inventory of MCP servers configured across supported apps, such as Claude Code, Claude Desktop, Gemini CLI, OpenCode, Codex and Cursor, and enforces allow or deny decisions on the device. For framework purposes, that discovered inventory is the third source behind component 1, and the per-device enforcement is what turns an acceptable-use clause into a control.

What runtime controls cannot do

Runtime enforcement is necessary but not sufficient. A gateway cannot tell whether a hiring use case is lawful, whether an impact assessment was honest, or whether the person assigned to oversight has real authority. It does not evaluate model quality, and it cannot see AI used on personal devices outside management. Those remain review-time and organisational controls. The argument is not that infrastructure replaces governance, but that governance without infrastructure produces paperwork rather than control.

A 90-day plan to stand up the framework

Consider a 3,000-person software company that sells into the EU, uses three model providers across a dozen internal applications, and has engineers running coding agents. A realistic first quarter looks like this.

Days 1 to 30: see and own.

1. Appoint an executive owner and a governance council with security, legal, data, platform and one business representative.
2. Build the first inventory from all three sources: a registration form for declared systems, provider invoices and API logs for observed usage, and an endpoint scan for installed AI apps and MCP servers.
3. Adopt the AI RMF function names and the AI 600-1 risk list as the shared vocabulary, so risk discussions use the same twelve categories.

Days 31 to 60: tier and enforce.

1. Assign every inventory entry a risk tier and a system owner. Flag anything that falls into EU Annex III categories for a full impact assessment using ISO/IEC 42005 as the method.
2. Put a gateway in the path for internal applications. Replace shared provider keys with one scoped key per application or team, set model allow-lists and budgets, and turn on secrets and PII guardrails.
3. Give agents their own keys with deny-by-default tool lists, and define which action types need human approval.

Days 61 to 90: evidence and respond.

1. Extend routing to managed endpoints, starting with a volunteer group, and switch pending apps and MCP servers from allowed to blocked once the approval queue is under control.
2. Write the incident runbook around the kill switch: who can revoke a key, who decides on Article 73 reporting, and how after-action reviews feed the policy.
3. Produce the first mapping document: each control with its NIST subcategory, ISO/IEC 42001 control and EU AI Act article. That document is the backbone of a later 42001 certification.

By day 90 the company will not be certified, but it will be able to answer the four governance questions with data rather than assertions. For teams choosing products for each layer, GenAI Brief’s separate roundup of AI governance tools compares the options.

Counter-arguments and the limits of the evidence

Three objections deserve a serious answer.

“Frameworks are paperwork; ship controls and skip the framework.” Controls without a framework drift. Without a risk tier, nobody knows which guardrails are mandatory for which system; without an owner, nobody reviews the logs. The framework is what tells the infrastructure what to enforce.

“The EU AI Act was delayed, so there is time.” Only the high-risk dates moved. Prohibited practices and AI literacy have applied since February 2025, transparency obligations since August 2026, and general-purpose model duties since August 2025. More practically, building inventory and logging takes quarters, and the Annex III date of 2 December 2027 is roughly fourteen months away.

“NIST is being revised, so wait.” NIST says the AI RMF is being revised but has published no successor. The four functions and the inventory, tiering and incident outcomes are stable concepts that every other framework shares; a revision will change citations, not the work.

There is also a limit to the evidence. No public study yet measures whether organisations with runtime enforcement have fewer AI incidents than those relying on policy and training. The case for runtime control rests on the structure of the problem, that agents decide at run time, rather than on outcome data. That is a reason to measure it inside the organisation: track guardrail triggers, blocked tools and revoked keys, and review them alongside incidents.

The judgement

We think the right way to build an AI governance framework in 2026 is to treat NIST AI RMF as the vocabulary, ISO/IEC 42001 as the operating system and the EU AI Act as the strictest requirement set, and to map all three onto one set of seven components. The component that decides whether the framework is real is not the policy or the committee but the control in the request path. Start with the inventory, put every model and tool call behind scoped identities with budgets, guardrails and logs, extend that to the laptops where agents now run, and let the evidence those controls produce drive the next round of policy.