Answer in brief
To evaluate an AI service, run it on your own real cases against acceptance criteria written in advance. Then get five answers in writing: where your data is kept and for how long, what a finished task costs with human checking included, what happens when the system is wrong or down, what independent evidence backs the security claims, and how you leave. Buy only after a short pilot with a stop rule.
A demo runs on the vendor’s cases, and yours are different
Every AI service looks capable in a demonstration. The inputs were chosen by the people selling it, the awkward cases were left out and nobody counted the attempts that failed before the meeting. None of that is dishonest. It is simply not information about how the service will behave on your documents, your customers and your exceptions.
The voluntary AI Risk Management Framework from the US National Institute of Standards and Technology (NIST) puts it in one sentence: accuracy measurements should always be paired with clearly defined and realistic test sets that are representative of conditions of expected use. For a buyer, performance claimed anywhere other than on your own material is a hypothesis.
This guide is a way to evaluate an AI service on the buyer’s terms, in seven checks: a trial on your own cases, data handling, the cost of a finished task, behaviour on failure, evidence of security, the exit, and a pilot with a stop rule. Most are questions that cost nothing to ask, and a vendor’s willingness to answer in writing is itself a result.
Before the trial, write down the problem and what good looks like
Start one step earlier than the product. The UK Information Commissioner’s Office (ICO) says that deploying an AI system to process personal data should be driven by evidence that there is a problem and a reasoned argument that AI is a sensible solution to it, not by the mere availability of the technology. The UK government’s Guidelines for AI procurement, published in 2020 for central government departments, tell buyers to provide a clear problem statement instead of a detailed specification.
So write three lines before you speak to anyone. What task is being done today, by whom, and how long does it take? What would count as done well enough by a machine? What must never happen? The third line matters most: a drafting tool that sometimes writes a clumsy sentence is an annoyance, and one that sometimes invents a delivery date is a liability.
The same guidelines list “defining acceptable model performance” among the practices to look for in a supplier. Do it yourself, in advance and in plain words: a reply is acceptable, for example, if a senior colleague would send it without changing the facts. Criteria written after the results are in tend to match the results.
Run the service on your own work, awkward cases included
Collect real examples of the task: ordinary ones, difficult ones and a few where the correct response is to decline or to pass the matter to a person. Remove personal data the trial does not need. There is no standard number of cases; the right size is the one at which a single odd result cannot swing the verdict.
Have the people who do the job today score the outputs against your criteria, without knowing which supplier produced which if you are comparing several. Count three results separately: acceptable as delivered, usable after correction, and wrong. The middle category is where the real cost of the service hides, because every correction is paid time.
The procurement guidelines describe proofs of concept as a way to test a technology before the solution goes live, and name “testing the model under a range of conditions” as a mark of a sound supplier. Ask for that range: run the same cases on different days, try longer inputs and poorer scans, and note whether the answers stay stable.
Where your data goes, how long it stays and who else can use it
Ask the data questions before the price, because the answers can end the conversation. Where is the data processed and stored? How long are inputs, outputs and logs kept? Are they used to train or improve models, and can the contract exclude that? Which subcontractors receive the data? What is deleted when you leave, and how is the deletion confirmed?
The ICO’s guidance on AI explains, in UK data protection terms, why these are your questions and not only the vendor’s. An organisation that decides what personal data is processed, for what purpose and how long it is retained is a controller, however a contract describes it. The guidance tells buyers to specify their requirements at the procurement stage, and says that if you cannot assess whether a third-party system would be compliant, good practice is to opt for a different one.
Two cautions keep this honest. The ICO guidance deals with UK data protection law, and its pages carry a notice that it is under review following the Data (Use and Access) Act, so read the live version; other countries have their own rules. If the service will handle customer or employee data, take the vendor’s written answers to your privacy adviser before you sign.
Price one finished task, not a seat or a thousand tokens
Price lists for AI services use units that suit the seller: seats, credits, requests, tokens. Your unit is a finished task, meaning an output that was accepted and used. To get from one to the other, take the trial results and add what the price list leaves out: discarded attempts, the minutes people spent checking and correcting, the set-up work, and the monthly hours of keeping the service working.
An illustration with invented round numbers shows the arithmetic. A month of use costs 300 in fees and produces 1,000 drafts, of which 800 are accepted. Staff spend 40 hours checking and correcting at 25 an hour, which is 1,000. The total is 1,300 for 800 finished tasks, about 1.63 each, while the fee alone suggested 0.30. These figures are an example, not market prices; the lesson is the gap between the two.
Then put the manual process through the same sum, with its own errors and rework, and compare like with like. The procurement guidelines ask buyers to weigh whole-of-life costs, including ongoing support and maintenance. Ask the vendor how its prices and usage limits have changed over the past year.
Wrong, unsure or offline: what the service does on a bad day
Every system fails, and what you are buying is the manner of failure. Ask to see what happens when the service does not know: does it say so, or does it produce a confident answer anyway? The NIST framework expects a system’s knowledge limits to be documented and expects the system to fail safely when made to operate beyond them.
Next comes the route to a person. When the system gives up or a customer objects, who receives the case, with what context, and how quickly? The framework lists processes for human oversight, channels through which users can report problems and appeal outcomes, and assigned responsibility for deactivating a system whose results are inconsistent with its intended use. In a small company that can be one named person and a switch.
Last, the outage. NIST asks for contingency processes to handle failures or incidents in third-party AI systems that an organisation treats as high-risk. Decide how the task gets done on a day when the service is down, and keep that manual route in working order. Ask the vendor for its availability record and incident notification terms, and read what the contract offers when it misses them.
Security and compliance are shown in documents, not adjectives
“Enterprise-grade”, “secure by design” and “fully compliant” are descriptions. Evidence has an author, a scope and a date. Ask for the most recent independent audit or assessment report and read what it covered, the summary of the last penetration test, the list of subcontractors, the incident history, and the document that describes the system’s limits and the conditions under which it was tested.
The frameworks point the same way. NIST expects security and resilience to be evaluated and documented, and expects independent assessors, or internal experts who were not the system’s front-line developers, to take part in regular assessments. The procurement guidelines suggest that buyers consider requiring independent audits and look for robust and proportionate security provision: a tool that rewrites public product descriptions needs less scrutiny than one that reads customer files.
Be exact about what a reference to a framework means. The NIST framework is voluntary, and its own text says that its actions are not a checklist and that organisations may select from among its categories. So “aligned with NIST” is a claim to unpack: ask which practices are in place and where they are written down. NIST published version 1.0 in January 2023, and when this article was checked in October 2026 its website said that version was being revised, so confirm the current edition before quoting its numbered subcategories.
Read the exit clause first: lock-in is agreed on signing day
Lock-in is rarely a trick. It accumulates: prompts tuned to one product, documents converted to one format, staff trained on one screen. The procurement guidelines tell buyers to avoid black-box systems and vendor lock-in, and name open standards as good practice.
Ask what you can take with you and in what form: your documents, the instructions and configurations written for you, your test cases and results, the logs. Ask who owns what was created during the engagement and what happens to your data on the vendor’s systems after termination. The guidelines advise defining end-of-contract roles and processes for both sides, and the ICO says a buyer should be able to switch provider if use of an outsourced service is no longer compliant.
Change is the other half of exit. The model behind a service will be updated, and behaviour you tested may shift without any action on your part. Ask how much notice you get before a model or a price changes, whether you can stay on the previous version for a period, and whether you may rerun your own test cases before the change reaches your customers.
A paid pilot with a stop rule, then a date for the next test
Bring the checks together in a short paid pilot. Put the acceptance criteria, the test cases, the duration and the decision date on one page, and agree in advance what result means stop. The procurement guidelines recommend go/no-go decision points, and NIST describes the same moment as a determination of whether the system achieves its intended purposes and whether deployment should proceed. A pilot without a stop rule is a purchase paid in instalments.
Passing the pilot does not end the evaluation. The guidelines put it plainly: lifespan testing is required, not a one-time decision, and the buyer should establish with the supplier how the model will be monitored once deployed. Keep your test cases, add every real failure to them, and rerun the set on a fixed date and after every announced model change.
Some answers should end the conversation early. A vendor that will not run a trial on your material, cannot say where your data is kept, offers no route to a person when the system fails, or has nothing to say about leaving is telling you how the relationship will go. A supplier that answers all seven checks in writing, including the uncomfortable ones, has done a good part of your risk assessment for you.
Practical checklist
- Write down the problem, the acceptance criteria and the one outcome that must never happen before you contact a vendor.
- Assemble real cases, including difficult and unanswerable ones, and have the people who do the job score every output.
- Get written answers on data location, retention, training use, subcontractors and deletion, and show them to your privacy adviser.
- Calculate the cost of one accepted task with checking time included, and put the manual process through the same calculation.
- Agree the fallback route, the notice period for model and price changes and the export of your material before you sign.
Questions and answers
How long should a trial of an AI service last?
There is no standard length. It should be long enough to run your full set of cases at least twice and to see an ordinary week of real use. Duration matters less than preparation: with criteria and cases ready on the first day, a short pilot gives a clearer answer than a long one that begins with exploration. Fix the decision date before the pilot starts.
What should I ask an AI vendor about my data?
Five things, in writing: the countries and systems in which the data is processed and stored, the retention period for inputs, outputs and logs, whether any of it is used to train or improve models, the subcontractors that receive it, and what is erased when the contract ends. If personal data is involved, pass the replies to your privacy adviser. A vendor that cannot answer precisely has told you something important.
Is a vendor’s benchmark score a reason to buy?
It is a reason to shortlist, not to buy. A benchmark measures performance on someone else’s test set under someone else’s conditions. The NIST framework asks for performance to be shown in conditions similar to the setting where a system will be used, and for a buyer that means your documents, your customers’ questions and your definition of an acceptable result.
Can a small business do this without a technical team?
Yes. Most of the checks are questions and a spreadsheet: a list of real cases, a column for each supplier, three possible scores and a note of the time spent correcting. The colleagues who handle the task now are the best judges of the output. Outside help is worth paying for in two places: reading the data protection terms and reviewing the security evidence.
What are the warning signs when buying an AI service?
Refusing a trial on your own material, vague answers about data retention or training use, no description of what happens when the system fails, evidence that consists of adjectives, and a contract with no export or termination terms. Pressure to sign before the pilot ends belongs on the same list. One of these is a question to press; several together are a reason to walk away.