Assessing an AI vendor

Why vendor assessment is AI governance

Fernhill's inventory (Lesson 6) showed two systems that would send data to a hosted model, and one, the applicant screening helper, that's high-risk. Whether Fernhill buys a screening product or builds one on a model, someone else's software will make the first pass at every applicant. The frameworks are clear that this risk is still Fernhill's:

• NIST AI RMF, GOVERN 6: "Policies and procedures are in place that address AI risks associated with third-party entities", including contingency plans if a third party's system fails.
• ISO/IEC 42001, Annex A.10: allocate responsibilities with suppliers and customers, and make sure suppliers' services fit your approach to responsible AI.
• EU AI Act: deployers have their own duties (Article 26), and a deployer who substantially modifies a system, or puts its name on one, becomes its provider (Lesson 4).
• OMB M-25-22: US agencies buying AI must put testing rights, data ownership and protections against vendor lock-in into contracts (Lesson 5).
• The EU's model contractual clauses for AI procurement, updated March 2025, give public buyers ready-made clauses in two versions: one for high-risk systems, and a "light" one for the rest.

The questionnaire

Fernhill's questionnaire has fourteen questions in six areas. Seven are must-haves: fail one, and the vendor is out.

Why some of these matter more than they look:

• D1, training on your data: "anonymized data may improve our models" means résumés help train a product sold to others. Anonymizing free text reliably is hard, and applicants never agreed to it.
• S1, SOC 2 Type II, not Type I: a SOC 2 audit checks a company's security controls. Type I checks they're designed well on one day; Type II checks they actually worked over a period, usually six to twelve months.
• C1, change notice and pinning: a vendor that "continuously improves" its model can change your system's behavior overnight. Every test in Lesson 8 is void the moment the model changes. Pinning means you choose when to move to a new version.
• L2, oversight features: Lesson 9's planted items, evidence display and review logs have to be possible in the product. If they're not, the deployer can't meet Article 26.

Each answer comes with its evidence: a contract section, a report, a certificate. The vendors' answers are stored as [answer, evidence] pairs:

"C1": ["30+ days and pinning", null],
"S1": ["SOC 2 Type II", "report dated 2026-06"],

The checker

def assess(criteria, answers, vtype):
    score = total = 0
    fails, unproven, asks, ours = [], [], [], []
    for c in criteria:
        if vtype not in c.get("applies", [vtype]):
            ours.append(f"{c['id']} {c['question']}")
            continue
        total += c["weight"]
        answer, evidence = answers.get(c["id"], [None, None])
        ok = answer in c["pass"]
        if ok and evidence and evidence != "none written":
            score += c["weight"]
        elif ok:
            score += c["weight"] / 2
            unproven.append(c["id"])
            asks.append(f"{c['id']}: put \"{answer}\" in the contract, with evidence")
        else:
            if c["must"]:
                fails.append(f"{c['id']} {c['question']} -> \"{answer}\"")
            asks.append(f"{c['id']}: require \"{c['pass'][0]}\" (they said \"{answer}\")")
    return score, total, fails, unproven, asks, ours

How it works:

• Weights (1 to 3) set how much each question counts; the score is a percentage of what applies.
• Evidence: a passing answer with evidence scores full weight; without it, half, and the claim becomes a contract ask.
• Must-haves: a failed must-have makes the vendor a NO-GO, whatever the score. A vendor at 90% that trains on your applicants' résumés is still a no.
• Contract asks: every gap turns into a sentence for the negotiation.
• applies (added after the first run, below) skips questions that don't fit this kind of vendor, and lists them as Fernhill's own job.

The verdict: NO-GO on any must-have failure, GO WITH CONDITIONS if anything is claimed without evidence, GO otherwise.

The first run: the checker got it wrong

The first version had no applies: every question went to every vendor. Fernhill's three candidates:

• Brightlane Hire and Tessaro Talent, HR screening products: the vendor builds the tool, and Fernhill uses it.
• Quillstone AI, a hosted model API: Fernhill would build the screening helper on it.

(Your machine)

python3 -I vendor_check.py vendors.json

Quillstone's result:

== Quillstone AI: hosted model API (Fernhill would build the tool, and be its provider)
   score 23/34 (68%); verdict: NO-GO
   must-have failed: T1 Independent bias audit with impact ratios, for hiring use? -> "not applicable: general model"
   must-have failed: C1 Notice before material model changes, and version pinning? -> "dated model versions; 6 months notice of retirement"
   must-have failed: L2 Oversight features: evidence shown, override, exportable logs? -> "not applicable: API"
   contract asks:
     - T1: require "independent, yearly" (they said "not applicable: general model")
     - C1: require "30+ days and pinning" (they said "dated model versions; 6 months notice of retirement")
     - L1: require "provider; plan by Dec 2027" (they said "general-purpose model provider; Code of Practice signatory")
     - L2: require "all three" (they said "not applicable: API")

Read the failures and they're not failures:

• T1 and L2 can't be the API vendor's job. A general model has no hiring use to audit and no review screen to show. If Fernhill builds the screening helper, the bias audit and the oversight features are Fernhill's to build and prove.
• C1 was a good answer, worded differently. "Dated model versions" is pinning: you call a specific dated version, and it doesn't change under you. Six months' notice before one is retired beats 30 days. The checker wanted an exact phrase.
• L1 was the right answer for this kind of vendor. A general-purpose model provider's EU AI Act duties are the model-level ones (documentation, copyright policy, training summary; Lesson 4), and signing the Code of Practice is how most show it.

A questionnaire written for one kind of vendor misjudges another. It's the most common mistake in vendor assessment: one long spreadsheet sent to everyone, scored by matching words. Two fixes went in: each question says which kinds of vendor it applies to, and passing answers include the right answer for each kind.

The second run

Use case: Applicant screening helper: summarizes each applicant's evidence against job criteria for HR (high-risk, Annex III employment)

14 criteria; must-haves: D1, D2, S1, T1, T2, C1, L2

== Brightlane Hire: HR screening software (they'd be the provider; Fernhill the deployer)
   score 31.5/34 (93%); verdict: GO WITH CONDITIONS
   claimed without evidence: C1, L1
   contract asks:
     - C1: put "30+ days and pinning" in the contract, with evidence
     - L1: put "provider; plan by Dec 2027" in the contract, with evidence

== Quillstone AI: hosted model API (Fernhill would build the tool, and be its provider)
   score 28/28 (100%); verdict: GO
   Fernhill's job, not theirs: T1 Independent bias audit with impact ratios, for hiring use?
   Fernhill's job, not theirs: L2 Oversight features: evidence shown, override, exportable logs?
   contract asks:
     - none

And the third vendor:

== Tessaro Talent: HR screening software (they'd be the provider; Fernhill the deployer)
   score 6.5/34 (19%); verdict: NO-GO
   must-have failed: D1 Is our data used to train or improve your models? -> "anonymized data may improve our models"
   must-have failed: D2 How long are inputs and outputs kept? -> "for the life of the contract"
   must-have failed: S1 Current SOC 2 Type II or ISO/IEC 27001? -> "SOC 2 Type I"
   must-have failed: T1 Independent bias audit with impact ratios, for hiring use? -> "internal fairness testing"
   must-have failed: C1 Notice before material model changes, and version pinning? -> "continuous improvement; no pinning"
   claimed without evidence: D3, T2, T3
   contract asks:
     - D1: require "no" (they said "anonymized data may improve our models")
     - D2: require "0 days" (they said "for the life of the contract")
     - D3: put "US" in the contract, with evidence
     - D4: require "yes" (they said "on request")
     - S1: require "SOC 2 Type II" (they said "SOC 2 Type I")
     - G1: require "ISO/IEC 42001 certified" (they said "responsible AI principles")
     - T1: require "independent, yearly" (they said "internal fairness testing")
     - T2: put "yes" in the contract, with evidence
     - T3: put "yes" in the contract, with evidence
     - C1: require "30+ days and pinning" (they said "continuous improvement; no pinning")
     - C2: require "72 hours or less" (they said "promptly")
     - L1: require "provider; plan by Dec 2027" (they said "working with counsel")
     - X1: require "yes, within 30 days" (they said "yes")

What each result means:

• Tessaro is out, five must-haves over. Its answers are a catalogue of soft language: "promptly", "on request", "working with counsel", "responsible AI principles" backed by a blog post. Its oversight features were shown in a demo and are probably real; it doesn't matter.
• Brightlane is a GO with conditions. Two claims have no evidence: change notice (no contract section) and its EU AI Act plan ("none written"). Both become contract clauses before signing.
• Quillstone scores 100%, and that's the trap. It's an excellent model vendor. But choosing it means Fernhill builds the screening helper, and two must-haves just moved onto Fernhill's own list.

The real decision: buy or build

Enlarge (opens the diagram full size)

Close

For a small company like Fernhill, the honest comparison isn't 93% against 100%. It's this:

• Buy from Brightlane: Fernhill is a deployer. It verifies Brightlane's claims, negotiates the two missing clauses, trains its reviewers, keeps logs, tells applicants, and monitors outcomes. Brightlane carries the provider duties: conformity assessment, technical documentation, the bias audit.
• Build on Quillstone: Fernhill is the provider of a high-risk system, with everything Lesson 4 listed: risk management, data governance, documentation, logging, oversight design, accuracy and robustness testing, a quality management system, conformity assessment and registration, by December 2, 2027 for an Annex III use. It also gets full control: Lessons 7 to 9's guard, tests and review design, exactly as Fernhill wants them.

Neither is wrong. But "the vendor passed" answers a different question from "can we carry what this choice puts on us?" For a small company with one high-risk use, buying from a vendor that's already doing the provider work, and holding it to that in the contract, is usually the realistic choice.

What goes in the contract

Every gap and every unproven claim becomes a clause. For a high-risk AI system, Fernhill's list:

Spot the problem

Work out what's wrong before you read the explanation.

"They're ISO 42001 certified, so we're covered." A company buys an AI tool that ranks loan applications. The vendor holds an ISO/IEC 42001 certificate and a SOC 2 Type II report, and the procurement team marks the AI risk review complete. A year later, a regulator asks the company for its own assessment of the tool's effect on applicants, and for evidence that its loan officers' review is meaningful. The company has neither. What's wrong? A vendor's certificates describe the vendor's management system, not your use of their product, and not your duties. ISO/IEC 42001 shows the vendor runs an AI management system; it doesn't certify that a specific product is fair, or fit for your use. And as the deployer of a high-risk system, the company has its own duties: oversight, monitoring, logs, informing people, and for credit scoring under the EU AI Act, a fundamental rights impact assessment. Three fixes: read what the certificate's scope actually covers; assess the use, not just the vendor, with your own tests (Lesson 8); and list your own duties as deployer and assign each one to someone.

Where you are now

You can assess an AI vendor with must-haves and evidence, spot soft answers, fit the questions to the kind of vendor, and see how the choice between buying and building moves duties onto you. The last lesson pulls the whole track together into the one document most organizations start with: an AI use policy.

Check yourself

Key terms

• Third-party risk: risk from systems and services you rely on but don't control.
• Must-have: a requirement no score can make up for; failing it rules a vendor out.
• Evidence: a document that backs a claim: a contract section, audit report or certificate.
• SOC 2 Type I and Type II: audits of a company's security controls, at one point in time (Type I) or over a period (Type II).
• Version pinning: calling a fixed model version, so its behavior can't change without your choice.
• Subprocessor: a company your vendor relies on to process your data, such as its model provider.
• Model contractual clauses (MCC-AI): the EU's template clauses for buying AI, for high-risk and other systems.
• Lock-in: dependence on a vendor that makes leaving costly or slow.

Go deeper

• Covington: the EU's updated AI model contractual clauses (opens in a new tab), and the EU's public buyers community page (opens in a new tab).
• NIST AI RMF, GOVERN 6.1 (opens in a new tab), on third-party AI risk.
• OMB M-25-22 (opens in a new tab), on how US agencies buy AI.
• Try: add a fourth vendor whose answers are all good but whose evidence is all "available under NDA". What should the checker say?
• Try: write the questions you'd send a model API vendor that you wouldn't send an HR software vendor.