The Vendor BS Detector

What this is for

You're evaluating an AI tool or vendor for your team and the sales deck looks great, which tells you nothing. This is for the meeting itself: the questions that separate a vendor who's built something real from one reselling someone else's model with a markup and a slide template.

Before you start

Know your own risk profile before the call. Which of these matters most to you: data leaving your control (regulated industry, client confidentiality), cost surprises (tight budget, long contract), or vendor failure (this tool would sit in a critical workflow)? Pick your top three questions from the list accordingly. You will not get honest answers to twelve questions in one meeting; you'll get honest answers to the three you press on.

The 12 questions

1. Where does our data go, and is it used for training? Good answer: names the specific model provider, states in writing whether your data trains any model (it shouldn't, by default), and can show you the data processing agreement on request. Bad answer: "It's all secure and private" with no specifics, or hesitation when you ask for it in writing.

2. What happens on your worst-case error? Good answer: a real story. A specific failure, what it cost, what changed afterward. Every vendor has one. Bad answer: "We haven't really had issues" or "Our error rate is extremely low." No tool with real usage has zero failures.

3. Is this a real model or a wrapper on someone else's? Good answer: straightforward. "We use OpenAI's/Anthropic's API and our value is in [specific thing: workflow, data, interface]." Being a wrapper isn't automatically bad if the wrapper adds real value. Bad answer: vague deflection, or claiming proprietary AI they clearly didn't build, with no technical detail when pressed.

4. What's the actual accuracy on our data, not your demo data? Good answer: offers a pilot on your real, messy data before you sign anything, and is upfront that accuracy on your data may differ from the demo. Bad answer: only shows you their curated demo and resists letting you test with your own inputs.

5. Who owns the outputs, and will you stand behind them? Good answer: clear, in writing: you own what the tool produces for you, and the contract includes indemnification if the tool's output creates an IP problem. Bad answer: "We'd have to check the terms," or ownership language buried in a EULA nobody read.

6. What does month 13 cost, and what's usage-based? Good answer: gives you the renewal price now, and, critically, explains any usage or token-based component and what happens when you exceed the included volume. Usage overages, not renewal uplift, are where 2026 AI contracts surprise people. Bad answer: "Let's cross that bridge later," or a flat year-one price with a usage meter quietly running underneath it.

7. What's the exit and data-export path? Good answer: a documented process to export your data in a stated format with a schema, and a deletion commitment measured in days after you leave. Bad answer: no clear answer, or export only in a proprietary format that locks you into rebuilding elsewhere.

8. What does implementation actually require from my team? Good answer: a specific list. Hours, roles, systems to connect, a realistic timeline with a named point of failure if it slips. Bad answer: "It's plug and play" with no detail. Nothing enterprise-relevant is truly plug and play.

9. Who else in my industry is live on this, and can I talk to them? Good answer: a live reference you can actually call, not just a logo on a slide. Bad answer: logos with no offer to connect you, or references who turn out to be a two-week pilot, not a real deployment.

10. What happens when you swap the underlying model? Good answer: acknowledges that they upgrade or switch model providers, describes how they test that your results don't degrade, and commits to advance notice. Bad answer: surprise that you asked. Vendors change models routinely now; if they haven't thought about how that changes your accuracy, they haven't thought about you.

11. If this is an agent, what exactly can it touch? Good answer: a precise list of the permissions it needs in your systems (read email? send email? write to the CRM?), scoped to least access, with an audit log you can see. Bad answer: "It integrates with everything." An AI that acts on your systems is an employee you can't interview; the permission list is the job description.

12. Show me the paperwork. Not a question, a request. The artifacts a real vendor produces on request: a SOC 2 Type II report (not "we're SOC 2 compliant" on a slide), a data processing agreement, a sub-processor list (who else touches your data), a written zero-data-retention or retention-period statement, and the export format from question 7. A vendor who has these sends them within a day. A vendor who doesn't will schedule a call about it.

Send this before the demo

Forward this to the vendor ahead of the meeting. It changes the demo from theater into an audit, and how they respond to receiving it is itself a signal:

Ahead of our meeting, please come prepared to address:

1. Where our data goes, who the underlying model provider is, and
   whether any of our data is used for training (in writing).
2. Your worst production failure to date and what changed after it.
3. Accuracy expectations on our data vs. your demo data, and whether
   you offer a pilot on our real inputs.
4. Output ownership and IP indemnification terms.
5. Full pricing including any usage-based components, overage rates,
   and expected year-two pricing.
6. Data export format, schema, and post-termination deletion timeline.
7. Implementation requirements from our team, in hours and roles.
8. Two live customer references in or near our industry.
9. Your process and notice policy when you change underlying models.
10. If the product acts on our systems: the exact permission list it
    requires and what audit logging we get.
11. Documents to send in advance: SOC 2 Type II report, DPA,
    sub-processor list, and data retention statement.

We'll use the meeting to go deep on three of these, and we'll bring our
own sample data for a live run.

Score the meeting

For each question you asked, mark it pass, hedge, or fail. Scoring rule: two hedges on questions 1, 4, or 7 and you walk. Data, accuracy, and exit are the three you can't fix with a contract amendment later. Everything else is negotiable; those three are structural.

Run the demo yourself, with your own data

Before the contract, not after: bring your own messy input. A real client email, a real spreadsheet with your actual formatting problems, a real transcript with crosstalk. Ask the vendor to run it live, unscripted, in the room. Watch what breaks. A vendor who resists this, or insists on using only their prepared examples, is telling you the tool doesn't hold up outside the demo script.

(Anonymize your sample data first: swap client names for placeholders and round the numbers. The demo is still real; the exposure isn't. The One-Page AI Policy covers the team-wide version of this rule.)

Where this goes wrong

• Accepting "we take security seriously" as an answer to question 1. It isn't an answer. It's a sentence with no content. Ask for the specific data policy in writing.
• Skipping the reference call. It feels like extra work; it's the single fastest way to find out if month 13 support looks like month 1 support.
• Letting the vendor pick the demo data. If you don't bring your own, you're evaluating their best case, not your real one.
• Treating the paperwork request as adversarial. Real vendors are relieved when a buyer asks for the SOC 2 report; it means they're not wasting a sales cycle on someone who can't buy.

The 2-minute version

Forward the pre-meeting block above to the vendor. In the meeting, press hard on your three highest-risk questions (usually 1, 4, and 7), bring your own messy data, and apply the scoring rule: two hedges on data, accuracy, or exit, and you walk.

---

Want to know where you actually stand with AI? The free AI Readiness Assessment takes 10 minutes and gives you a personalized roadmap.

Weekly, on Tuesdays. The Gen Xcelerator: practical AI intelligence for people who already know how to do the job. Subscribe free.

Experience Is the API. GenXcelerate