An AI feature can give convincing answers in a demonstration and still struggle with everyday work. Before staff rely on it, agree what a usable result looks like, test realistic questions with someone who knows the job, and use the failures to decide what is ready.
Imagine a feature that prepares draft answers from approved internal guidance. A staff member reads each draft before using it. This is a hypothetical example, but it makes the acceptance question easier to see.
A prototype answers a few questions well. Now someone wants to put it in front of staff. Before you agree, ask the supplier to show how it handles the work those people will actually give it, including the questions it cannot answer.
You can make that conversation quite concrete. Choose one job, agree what a usable result looks like, and examine results from a set of realistic examples with someone who knows the work. That gives you something to base a release decision on.
Write down what the feature may use, who it is for, and what it produces. In this example, it drafts an answer from the guidance that person is allowed to see. Sending the answer or changing a record would be a separate capability to test.
Then agree what counts as usable. The draft should answer the question, match the approved material, preserve any conditions that matter, and show which source supports it. If the material cannot answer the question, the draft should say so. A clear admission of missing information can be a successful result.
A made-up rule, an omitted condition that changes the answer, or a confident answer based on the wrong guidance would make the draft unusable. Agree these distinctions before reviewing results. Otherwise, a fluent answer can be surprisingly persuasive.
For the draft-answer example, include at least these kinds of situation:
The guidance contains a clear answer. Does the draft use it accurately and point to the right source?
Two procedures could apply. Does the feature ask for the detail needed to choose, rather than quietly choosing one?
The approved guidance does not cover the request. Does the feature explain the gap and leave the answer for a person to resolve?
Those examples are a starting point. Build the test set around the real mix of work: common requests, incomplete information, difficult wording and mistakes whose consequences matter. Use authorized representative data or safely prepared examples, and keep confidential records out of unapproved services.
What would you test in your AI feature?Tell us what staff should use it for. We can discuss the intended job, where a person should stay involved, and what you would need to check before using it.Discuss your feature
Microsoft's evaluation guidance recommends representative interactions and challenging cases.
For a manager, the useful question is whether the examples cover the situations staff need help with. A general model benchmark cannot settle that question for your application.
Ask a person familiar with the task to review the answers against the agreed criteria. They should be able to explain why an answer is usable, wrong or uncertain. A second look helps where a judgement is disputed or the consequences of a mistake are significant.
The voluntary NIST AI Risk Management Framework supports documented tests, conditions resembling deployment and input from relevant experts.
It gives a useful basis for asking what was tested and who checked it. It does not supply a universal pass mark for your feature.
Keep some evaluation examples separate from the examples used to improve the feature. If the supplier keeps adjusting it against every test question, you learn how well it handles those familiar questions. Separate examples give you a better check on how it handles work it has not been tuned around.
Automated scoring can help sort results, but its judgements need checking too. People can miss mistakes, and an AI used to grade another AI can also be wrong. Ask to see actual answers and the reasons behind the scores.
An overall score can help compare versions. Alongside it, ask to see results grouped by the situations and mistakes that matter. Ordinary questions may go well while missing-source questions repeatedly produce invented answers. The average hides a decision you need to make about that particular use.
Record which failures happened, how serious they were, and what remains uncertain. Decide which ones prevent the intended use. The acceptable error depends on the task and what happens after a mistake; there is no percentage that settles this for every business.
If the feature also takes actions, check what actually happened in the connected system. For example, if it says it updated a record, inspect whether the correct record has the intended value.
Anthropic's January 2026 account of evaluating AI agents explains why the final outcome matters when software acts.
A message saying the job is complete does not establish that it succeeded.
Output quality is one part of acceptance. Security, permissions, privacy and the rest of the workflow need their own appropriate checks. A good answer score cannot establish that those parts work.
Ask the supplier for a short record of the task and limits, test cases and expected results, who reviewed them, important failures, remaining uncertainty, and the version tested. Include the model, instructions, source material and connected software where they affect the result. Agree which checks must run again after changes to any of them.
That evidence should support one of three decisions:
The results support a defined task within stated limits. Staff know when to review, stop or fall back to the existing process.
Specific failures need correction. Agree which cases must be rerun and which separate examples will check whether the fix holds.
Important failures remain unresolved, or the evidence is too thin to support the proposed use.
A limited pilot can add evidence from actual work. Set its users, tasks, permissions, review arrangements and fallback in advance, with someone responsible for watching problems and pausing it. Good results from a small pilot support that limited use; they cannot prove general reliability.
It answers the question using the approved material, keeps any conditions that affect the answer, and shows the supporting source. When the material cannot answer the question, clearly identifying that gap can be a usable result too.
There is no percentage that settles this for every business. Look at which questions failed, how serious the mistakes were, and what happens after an error. A good overall score can still hide a failure that prevents the intended use.
Someone who knows the work should check actual answers against criteria agreed before testing. Get a second review when a judgement is disputed or the consequences of a mistake are significant. Automated scores can help, but their judgements also need checking.
Agree which checks must be repeated when the model, instructions, source material or connected software changes. Keep some examples separate from those used to improve the feature, and use problems found during a limited pilot or everyday work to inform further testing.
You should leave the acceptance conversation knowing what staff can rely on, where a person still needs to decide, and what would trigger another test. If you are planning an AI feature in a custom business application, we can discuss the intended job and the evidence you would need before accepting it.
Last newsletter: making more money from a subscription site or app you already own.
The one before: how to spot AI nonsense from real success stories.
Interested?
Weekly Or just let us know and we'll adjust it for you. Sounds good?
