What AI governance is, in engineering terms

What AI governance is

AI governance is how an organization makes sure the AI it builds, buys and uses does what it's meant to, without harming the people it touches, and that someone can show it. In engineering terms, that's six things:

1. Know what you have. Every AI system and AI feature, what it's for, who owns it. (An inventory: Lesson 6.)
2. Decide what could go wrong, and how badly, for whom. (Risk: Lessons 2 to 5.)
3. Control what goes in. Which data a model may see, and what must never go in a prompt. (Lesson 7.)
4. Check it before you trust it, and keep checking. (Testing and red-teaming: Lesson 8.)
5. Keep a person where it matters. Someone who can see what it did, and overrule it. (Human oversight: Lesson 9.)
6. Keep records, and a plan for when it goes wrong. What it did, when, on what, and who decided.

If that sounds like the Compliance and Security tracks, it is: access, logging, change management, testing and incident response, applied to systems that behave in new ways. Governance adds the parts about purpose and people: should this use AI at all, and who's affected if it's wrong?

A home comparison, as in the Compliance track: a family that hires a nanny doesn't need a regulation to know they should check references, agree what the nanny may and mayn't do, keep the important decisions for themselves, and notice if something seems off. AI governance is those habits, written down and applied to software that makes judgments.

Why AI needs governing differently

Ordinary software does what its code says, every time. A model doesn't quite:

• It's probabilistic. The same question can get different answers, and a confident one can be wrong.
• It's hard to inspect. You can read a function; you can't read why a model chose a word.
• It's shaped by data you may never see: what it was trained on decides what it knows, gets wrong, and leans towards.
• It changes under you. A provider updates a hosted model, and behavior shifts with no change on your side.
• It can be talked into things. Text it reads can carry instructions ("prompt injection"), and the model can't always tell data from orders.
• People over-trust it. A fluent answer looks right; a person reviewing hundreds of them starts approving without reading (automation bias).

None of those are reasons not to use AI. They're the reasons each of the six habits above matters more than it would for a spreadsheet macro.

The rules and frameworks, as of November 2026

Four names come up in nearly every conversation. They do different jobs:

Two more things to know about the US. There's no federal AI law: the 2023 executive order on AI was revoked in January 2025, and federal policy since has favored speed and adoption. A December 2025 executive order set out to challenge state AI laws, and how far that reaches is still being fought over, so check before relying on any state's rules. Lesson 5 goes through it properly.

One more change, and you'll see it everywhere: since September 29, 2026, Executive Order 14434, "Inaugurating the Era of Super Intelligence", has federal agencies say "super intelligence" (SI) instead of "artificial intelligence" in their own documents and websites. The legal meaning hasn't changed, and older documents keep their names, so NIST's pages now talk about SI while its framework is still the AI RMF. This track says AI, as most of the world still does.

How they fit together: NIST tells you how to think about AI risk; ISO 42001 tells you how to run that thinking as an auditable system; the EU AI Act tells you what you legally must do if you're in its reach; OMB's memos tell federal agencies what they must do. An engineer usually meets them as the same requests: an inventory entry, a risk assessment, test results, a named owner, and evidence of human review.

A real system, mapped

CivicGrid is a data service covering about 3,000 US cities. Its weekly mayor check keeps the mayors' names current: it reads each city's own web page and compares the name there with the one in the database. A small language model, running on a home server, does one narrow job in the middle of it.

Here's the system against the questions governance asks. Everything below is how version 7 of the check works, from its own documentation.

What's it for, and who's affected?

Purpose: catch mayors who've changed, so the data stays right. Affected: anyone using CivicGrid's data, and, indirectly, the mayors themselves: a wrong name attributes an office to the wrong person. The harm from one mistake is small and fixable; the harm from many, unnoticed, is a data service nobody trusts.

What does the model actually do?

Much less than you'd expect. qwen2.5:7b, running locally with Ollama, reads one name off a page. Its own documentation says "extraction only, no judgment". Everything else is ordinary code, before and after the model:

1. Fetch the city's page, with an identified user agent, honoring robots.txt.
2. Domain gate: after redirects, the page must be on the city's own website, or a .gov or .<state>.us address carrying the city's name. Anything else gets no model call at all.
3. Unchanged? Stop. A hash of the page's text is compared with last week's; no change, no model call.
4. State gate: the page must still name the city's state.
5. The model reads one name.
6. Verbatim guard: the name must appear, letter for letter, on the page. A name the model invented fails here.
7. Compare with the current mayor's name, allowing for nicknames, and needing a first and last name to match.
8. A match stamps the mayor as verified, with the page, model and run recorded. A difference becomes a proposal, and nothing else.

What data goes in?

Public web pages, and nothing else: no personal data beyond what a city publishes about its own officials, no customer data. The model runs on hardware CivicGrid controls, so nothing is sent to a model provider. (Lesson 7 is about this question for systems where the answer is harder.)

Where's the person?

The model never changes a name. The only write it can cause is the "verified" stamp on an exact match. Every difference waits at an admin review page, /admin/review, for a person to approve or reject. And a circuit breaker halts the whole run if too many cities propose a change at once: a website template rollout or a broken parser looks exactly like mass mayor turnover, and it never is.

What's recorded?

Every run gets a row, and its ID goes on every verification and every proposal, so any "verified" stamp can be traced to the page, model and run that produced it. That's the record an auditor, or the person investigating a complaint, would ask for.

How is it tested?

Three cities are regression cases: Bullhead City, Arizona; Lancaster, Ohio; and Wausau, Wisconsin. Any change to the check has to pass them first, in a dry run that writes nothing. (Lesson 8 covers why fixed test cases matter for a component that isn't deterministic.)

What happened when it went wrong?

Version 7 exists because of a mistake. Lancaster, Ohio was stamped "verified" with the mayor of Lancaster, Pennsylvania. Two cities, one name, and a page that matched closely enough. Nothing was changed (the check only stamps matches), but a wrong name was marked as checked, which is worse than unchecked: it looks trustworthy.

The fix was governance in miniature: find the cause, add controls, add a test so it can't come back. The domain gate (the page must be the city's own), the match records (so every stamp is traceable), a stricter name comparison, and Lancaster, Ohio as a permanent regression case.

What it doesn't have yet

An honest map includes the gaps. There's no written policy for what the model may be used for beyond this job, no formal risk assessment, and no measure of how often reviewers reject proposals (a rising approval rate with no rejections would be a sign of automation bias). The rest of this track adds those.

Map one of your own

Pick one AI use you know: a chatbot your team uses, a coding assistant, an AI feature in a product you build, a model in a pipeline. Answer these, one line each:

If you can't answer one, that's the finding. Most teams can't answer the first two for the AI tools their own people use day to day.

Spot the problem

Work out what's wrong before you read the explanation.

"We don't use AI." A company's leadership is asked, for a customer's security questionnaire, whether it uses AI in handling customer data. Engineering says no: the company hasn't built or bought any AI product. The answer goes out as "No". A month later, the customer finds their account details in a support ticket summary written by an AI feature in the help desk software, switched on by the vendor in an update, and in a chatbot conversation a support agent used to draft a reply. What went wrong? Nobody knew what AI was in use, because nobody had looked. AI arrives in three ways besides building it: features vendors switch on in tools you already pay for, tools people use on their own (often called shadow AI), and models inside other services. "We didn't build or buy any" answers none of those. Three fixes: inventory what's actually in use (vendor features, browser extensions, chat tools, not just projects), the subject of Lesson 6; set a rule for what may go into a prompt and make the approved tools easy to use, Lesson 7; and answer questionnaires from the inventory, not from memory.

Where you are now

You can explain AI governance as a set of engineering habits, say why AI needs them more than ordinary software, name the main frameworks and laws and where each stands, and map a real AI system against the questions a review would ask. The next lesson opens the most widely used of those frameworks, the NIST AI RMF, and its four functions: Govern, Map, Measure and Manage.

Check yourself

Key terms

• AI governance: how an organization makes sure the AI it builds, buys and uses does what it should, safely and lawfully, and can show it.
• AI system: software that uses a model to produce outputs, such as text, decisions or predictions, from inputs.
• Automation bias: the tendency to accept a machine's answer without checking, especially when reviewing many.
• Prompt injection: text a model reads that carries instructions, which the model may follow.
• Risk-based approach: scaling controls to how much harm a system could do, and to whom.
• High-impact AI: OMB's term for federal AI whose output is the basis for decisions with significant effects on rights or safety.
• Shadow AI: AI tools people use at work without the organization knowing or approving.
• Regression case: a fixed test that a past failure can't come back.
• Human oversight: a person who can see what an AI system did and change or stop it.

Go deeper

• NIST: AI Risk Management Framework (opens in a new tab), and the Generative AI Profile (opens in a new tab).
• The AI Act Explorer's Digital Omnibus page (opens in a new tab), for the EU AI Act's current dates.
• Gibson Dunn: EU AI Act Omnibus agreement (opens in a new tab), a readable summary of what moved.
• OMB M-25-21 (opens in a new tab), Accelerating Federal Use of AI through Innovation, Governance, and Public Trust.
• Try: fill in the eight-question table for an AI tool you used this week. Which answer did you have to guess?
• Try: find out which tools you pay for have switched on AI features in the last year. Who decided?