How to Evaluate an AI Vendor Without Being Fooled by a Demo
The 12-question technical due-diligence framework we hand to enterprise buyers — designed to separate genuine AI engineering from LinkedIn-grade theatre.
The industry has a demo problem
The average enterprise AI demo is a polished walkthrough of a hand-curated happy path against a hand-curated dataset. It tells you nothing about whether the vendor can ship. This framework is what we hand to enterprise buyers to cut through the theatre in three meetings, not three months.
1. Show me the eval harness
Ask to see the evaluation code that runs before every deploy. Ask what breaks it. If the vendor doesn't have a golden set, doesn't run automated evals, or measures only accuracy on a demo dataset — they will not survive contact with your production data.
2. Show me a failing case
Any AI vendor with production experience knows exactly how their system fails and can walk through a real failing case, the root-cause investigation and the fix. A vendor who claims their system 'doesn't really fail' has either not shipped it or is not paying attention.
3. Show me the audit log
For any regulated workload, ask for a sample audit trail: prompt, retrieved sources, model version, tool calls, response, user identity, timestamp. If the vendor's answer includes 'we can build that' — they haven't yet.
4. Show me the cost dashboard
Ask what a customer's monthly bill looks like, split by workload and by tenant. Vendors who can't answer this in one screen will surprise you with a five-figure overage in month four.
5. Show me the retry policy and rate-limit strategy
This is the boring engineering question that separates operators from prototypers. If the vendor can't describe their exponential-backoff strategy, per-tenant rate limits and circuit breakers on provider errors — expect outages during any provider incident.
6. Which model, and why?
'We use GPT' is not an answer. The right answer names the model per task, the fallback, the routing policy, and the last time they evaluated an alternative. Vendors locked into one model are locked into one vendor's roadmap.
7. What data leaves my perimeter?
Ask what data crosses to which providers under what agreements, what is retained, what is used for training, and what your data-processing addendum looks like. If the vendor waves this off, they are a compliance incident on your risk register.
8. Who owns the prompts?
Prompts, tool schemas, evaluation datasets — is this IP you own, or does the vendor keep it? A vendor who keeps the prompts owns your switching cost.
9. What's the human-in-the-loop pattern?
For any high-stakes workflow, ask where the human is, how they see the AI's reasoning, and how their corrections flow back. Full autonomy pitches for consequential decisions are red flags.
10. How do you handle version drift?
Foundation models update; behaviour changes. Ask how the vendor pins model versions, tests against new versions, and manages upgrade windows. 'We use whatever's latest' means your system's behaviour is not under your control.
11. Give me two customer references at your scale
Not testimonials — reference calls. Ask the references about incidents, cost surprises and what they wish they'd known. This one call is worth ten demos.
12. Show me the runbook
Ask for the operational runbook: what happens when the model returns garbage, when a provider goes down, when a tool call fails, when cost spikes. If the runbook doesn't exist, you are the runbook.
Key takeaways
- Ask for the eval harness, the audit log and the cost dashboard on first call
- Insist on real failing cases and how they were fixed
- Confirm data flow, retention and prompt IP ownership in writing
- Take two reference calls before signing anything
T7 Research
Enterprise AI Research Group
T7 Research is the research arm of T7 Solution, focused on benchmarking LLMs, evaluating RAG patterns, and compiling implementation playbooks for enterprise technology leaders.