There is an absurd number of agencies that have rebranded themselves as “AI agencies” in the last 24 months. Some of them have shipped real work. Many of them have shipped slide decks. If you're a small-business owner trying to figure out who can actually help, the marketing pages are a poor signal. The following ten questions are. Ask them all, and pay attention to the texture of the answers, not just the content.
1 · Have you shipped this in production at a business my size?
Big consulting firms have great case studies from enterprise clients. The lessons rarely transfer. An AI shop that has shipped work for SMBs your size (revenue, headcount, complexity) is in a different category from one that has done Fortune-500 engagements. Ask for two or three production deployments at businesses with 10–100 people and under $10M revenue. If they don't have any, you might be the prototype.
2 · Who owns the code?
If the answer is “we do, you license it from us,” understand what that means. You're renting the system that runs your business. If the relationship sours, or the agency goes out of business, what happens? Catalyst's default is that you own the code. We host it; you have the source, the database, and a clean exit. Ask explicitly.
3 · What's the lock-in story?
Beyond code ownership: what other dependencies are you taking on? A proprietary platform you can never leave? A consultant who is the only person who knows how the system works? A licensing arrangement that ratchets up after year one? The right time to ask about lock-in is before you sign, not after.
4 · Is the AI doing the work, or just talking about it?
A lot of “AI” offerings in this market are wrappers around a chatbot UI. That's fine if you want a chatbot. It's not the same as agents that actually do work in the background. Ask for a demo of an agent acting on production data. Watch it pick up a real input, take an action, and produce a verifiable output. If the demo is “here's our chat interface,” ask harder.
5 · How do you handle wrong outputs?
All LLM-based systems are wrong sometimes. The question is not whether they're wrong but how the system catches it and what happens next. Ask for the failure-mode story. Does the system flag low-confidence outputs for human review? Is there a draft-then-approve step for anything irreversible? Are there logs that let you audit decisions? “Our AI just doesn't get things wrong” is the wrong answer; it means they haven't shipped at scale.
6 · Hourly billing or fixed-scope?
Hourly billing is great for the vendor and terrible for the buyer. The vendor's incentive is to maximize hours; the buyer's is to minimize them. Fixed-scope (with clear acceptance criteria) puts everyone on the same side of the table. Ask what their engagement model is, and if they only do hourly, ask why. Sometimes it's a legitimate research-grade project. Often it's a vendor that doesn't want accountability.
7 · What does support look like after launch?
The cost of an AI deployment is not the build. It's the next three years of maintenance, drift correction, prompt updates, and new edge cases. Ask what the post-launch relationship looks like. Is there ongoing support? Is it staffed by the same people who built the system, or a tier-one ticket queue? How fast do they respond when something is broken?
8 · Can you give me three references I can actually call?
Not testimonials on the website. Three current clients, names and phone numbers, with permission to be called. If the agency can't produce three, that's signal. If they can, ask the references three things: did the project hit the agreed scope? Is the team responsive a year later? Would you hire them again?
9 · What does month 13 look like?
Most agencies will sell you a great month 1. The interesting question is month 13: a year in, is the system still doing what it was supposed to? Has it been kept up to date? Are there new things the team has learned to use it for? Or has it slowly degraded into a thing nobody trusts? Ask. Then check with the references.
10 · Will you tell me no?
Maybe the most important question. Will this agency say “you don't need this,” “a SaaS would be cheaper here,” or “you should fix your process before automating it”? The agencies that won't say no are selling you everything they have. The ones that will are partners. Ask outright what they've recently recommended against for a prospect. If they can't think of an example, they're not honest about scope.
The agencies that won't tell you no are selling you everything they have. The ones that will are partners.
The bonus question
One more, off the record: “What would the cheaper version of this project look like, and why aren't we doing that?” The answer should be specific and honest. If the cheap version is fine, the agency should say so. If it isn't, the agency should be able to articulate exactly why the bigger version is worth the difference. If neither answer comes easily, you're talking to a salesperson.
The honest part
This checklist is harder on us than it sounds. We use it internally when we audit our own engagements. The score isn't always perfect. But asking these questions out loud, and being able to answer them, is the difference between an agency that's serious about the work and one that's surfing the AI trend wave. Pick the first kind.
