15 Questions to Ask an AI Development Company Before You Hire or Sign, and the Red Flags (2026)

"AI development company" gets about 2,400 Google searches a month in the US, 210 in the UK and 110 in Australia, and "AI automation agency" gets 4,400, 720 and 390 (Google Ads data through DataForSEO, read 18 October 2026). The question buyers ask next, what to ask that company before signing, barely shows up on Google at all. People ask ChatGPT instead.

When we put the two buyer versions of that question to the ChatGPT app in the US, UK and Australia on 18 October 2026, 6 of its 13 web searches named the NIST AI Risk Management Framework, one named the OWASP Top 10 for LLM applications and two named the UK ICO. Google, by contrast, still reads the phrase as a job interview: 5 of the 8 US results for "questions to ask ai development company" were interview-question pages or interview software. This page is for the buyer before you hire or sign, and it puts the NIST, OWASP and ICO checks next to the plain business questions.

It works the same whether you are vetting a custom AI development company, an AI agent development company, an AI software development company, an AI chatbot development company, an AI app development company, an AI consulting agency, or an AI automation agency that builds workflows in tools such as n8n or Make. It suits a firm of 10 to 50 staff hiring its first AI vendor, a company of 50 to 500 running a formal selection, and a 500+ enterprise adding AI to an existing vendor risk assessment. If you are hiring one person rather than a company, read our guide on how to hire an AI agent developer instead.

The 15 questions at a glance

Four groups. Send them in writing before the first call, and keep the answers for your scorecard.

Proof: can they show it working?

1. Can you show me an agent you built that is running in production today?
2. Will you build a working demo on our data before we sign?
3. Who exactly will do the work, and can we meet them?
4. Tell me about a project that went wrong. What did you change afterwards?

Ownership and data

5. Who owns the code, the prompts and the evaluation set?
6. Whose accounts hold the model API keys and the cloud servers?
7. Are you our data controller or our processor, and will you sign a data processing agreement?
8. Will our data train anything of yours, or anyone else's?

Security and testing

9. How do you test for prompt injection (OWASP LLM01)?
10. What can the agent do without a person approving it (OWASP LLM06)?
11. Show us your test set and the pass mark.
12. What logs do we get, and where do they live?

After launch, and the money

13. What happens when the model provider retires the model you built on?
14. How is support priced, and is there a monthly minimum?
15. Is each phase scoped, priced and signed on its own?

What the guides that rank already ask, and what they leave out

On 18 October 2026 we read the buyer guides that Google ranks and the assistants cite for this question, then searched each page's text for OWASP, prompt injection, NIST, the ICO, the OAIC and "test set".

That gap matters because the assistants go looking for the standards. Across the 335 citations the five engines gave for these two buyer questions, NIST pages took 18 and ICO pages 10. The questions below keep the business checks the rankers share and add the security, testing and privacy-law checks they skip.

Proof: can they show it working?

These four answers sort real builders from resellers fastest. Gartner's 25 June 2025 press releaseestimated that only about 130 of the thousands of agentic AI vendors are real, and named the rest of the rebranding "agent washing".

1. Can you show me an agent you built that is running in production today?

Production means real users or real documents, every day, with someone accountable when it fails. A video or a sandbox does not count.

• A good answerThey share a screen into a live system (with client data masked), show a recent run, the log of what the agent read and did, and the step where a person approved or corrected it. They can tell you what the agent does when it is unsure.
• Red flagOnly a recorded video, a slide, or a demo on made-up data. Every past project is under a non-disclosure agreement and there is no masked version they can show.

2. Will you build a working demo on our data before we sign?

A small demo on a sample of your real records, say a few dozen past emails, orders or invoices, shows whether the team can cope with your mess. Slides cannot.

• A good answerYes, for one job, on a sample you choose, in a fixed short time. They tell you exactly which records they need, how they will hold them and when they will delete them.
• Red flagA demo only after a paid discovery phase, or a demo on generic sample data that looks nothing like your documents.

3. Who exactly will do the work, and can we meet them?

The UK government's Guidelines for AI procurement (8 June 2020) tell public buyers to require the supplier to assemble a team with the right skill sets. Private buyers should ask the same thing.

• A good answerNames and roles, including the person who will write the prompts and the integration code. That person joins a call before you sign. They say who takes over if someone leaves.
• Red flagOnly the sales lead appears. "Our team" has no names. Parts of the work go to a subcontractor you only hear about later.

4. Tell me about a project that went wrong. What did you change afterwards?

BotsCrew asks a version of this too: can the vendor name three reasons its last AI projects failed. The answer shows whether a firm learns.

• A good answerA specific project (no client name needed), the cause, such as messy source data or an approval step left out, and the process change that followed.
• Red flag"None of our projects has failed." Or a story where the client was the only problem.

Ownership and data: who holds what when the project ends?

An agent is the code, the prompts that steer the model, the evaluation set that checks it, the logs and the accounts that hold the model keys. Lose any one and switching vendors means starting again. NIST's Generative AI Profile, NIST AI 600-1 (July 2024), asks organisations to draft contracts and service level agreements that specify content ownership, usage rights, quality standards and security requirements (action GV-6.1-004).

5. Who owns the code, the prompts and the evaluation set?

An evaluation set is a list of real inputs with the answer you expect for each. It is how anyone checks the agent after a change, so it is as valuable as the code.

• A good answerYou own all of it, written into the contract, with the code in a repository in your own account from the first day.
• Red flagThe code is "licensed" to you, the prompts are the vendor's intellectual property, or the agent lives on a platform you cannot export from.

6. Whose accounts hold the model API keys and the cloud servers?

• A good answerYour accounts. The vendor gets access you can see and switch off. You can read the usage and the bills yourself.
• Red flagEverything runs in the vendor's accounts and you receive one bundled invoice. If you leave, the agent stops.

7. Are you our data controller or our processor, and will you sign a data processing agreement?

Under UK GDPR, the controller decides why and how personal data is used and the processor acts on its instructions. The ICO's guidance on AI accountability and governance(last updated 15 March 2023, under review) says that if a firm has no purpose of its own and only acts on a client's instructions, it is likely a processor. Its contract guidancelists the terms UK GDPR Article 28(3) requires in a controller and processor contract. The accountability guidance gives the example of a provider that trains its own model (likely a controller for that part) and then runs it on each client's data (likely a processor for that part).

• A good answerA clear role for each use of your data, a data processing agreement ready before the demo touches customer records, and a list of the model providers and other subprocessors that will see the data.
• Red flag"We are GDPR friendly" with no agreement, or a vague answer about which companies receive your data.

8. Will our data train anything of yours, or anyone else's?

• A good answerNo, in writing, plus the settings they use with each model provider so that your prompts and records are not used for training.
• Red flag"We may use anonymised data to improve our services." That is the line that, in the ICO's reasoning, can make the vendor a controller of your data.

Security and testing: how will they prove it behaves?

Most buyer guides stop at "ask about security". Better to ask about named risks. The OWASP Top 10 for LLM applications 2025, from the OWASP Gen AI Security Project, gives you the names. An LLM, or large language model, is the AI model behind tools such as ChatGPT and Claude.

9. How do you test for prompt injection (OWASP LLM01)?

OWASP defines prompt injectionas prompts that alter the model's behaviour or output in unintended ways, and notes the hidden text does not need to be visible to people. If your agent reads customer emails, supplier PDFs or web pages, someone can plant instructions in them.

• A good answerThey show test cases with planted instructions in documents and emails, explain what the agent is blocked from doing even if it is fooled, and rerun those tests before every change.
• Red flag"The model provider handles that." Model providers add their own guards, but your agent's tools and permissions are the builder's job.

10. What can the agent do without a person approving it (OWASP LLM06)?

OWASP calls this Excessive Agency and says its root cause is usually one or more of excessive functionality, excessive permissions or excessive autonomy. One of its listed fixes is to require a human to approve high-impact actions before they are taken.

• A good answerA written list of every action the agent can take, which ones need a person to approve (a refund, a message to a customer, a posting to the ledger) and the narrow account permissions it runs with.
• Red flagThe agent connects with an admin account "to keep it simple", or nobody can tell you what it is allowed to change.

11. Show us your test set and the pass mark.

A test set is a batch of real cases from your business with the expected answer for each. The pass mark is the score the agent must reach before it goes live or before any change ships. The FTC's Operation AI Comply(25 September 2024) shows why this matters: its complaint against DoNotPay alleged the company did not test whether its AI chatbot's output matched the level of a human lawyer, the expertise it claimed to stand in for.

• A good answerThey show a test set from a past project, agree with you how yours will be built from your own cases, and put the pass mark in the statement of work.
• Red flag"We tested it and it works well." Or a single accuracy number with no cases behind it.

12. What logs do we get, and where do they live?

NIST AI 600-1 action GV-6.1-008 asks organisations to keep records of changes third parties make to content, including sources and timestamps. For an agent that means a record of every input, tool call, output and approval.

• A good answerEvery run is logged in your own account, kept for as long as your policy says, and searchable by your team without asking the vendor.
• Red flagLogs exist only in the vendor's dashboard, are deleted after a few days, or cost extra to export.

After launch, and the money

13. What happens when the model provider retires the model you built on?

Models are retired on a schedule. Anthropic's model deprecations page says it regularly retires older models and that requests to retired models will fail. OWASP lists the third-party models and libraries an agent depends on under LLM03 Supply Chain, and NIST AI 600-1 action GV-6.2-007 asks buyers to consider system changes over time, such as drift and decay, when they review vendor contracts.

• A good answerThey watch the providers' retirement notices, rerun your test set on the replacement model before switching, and the contract says how that work is scoped and priced.
• Red flagNo plan, or the agent is tied to one model in a way that means a rebuild when it is retired.

14. How is support priced, and is there a monthly minimum?

Expect some mix of defects fixed inside the build price for a set cover period, changes quoted one at a time, and an ongoing support plan. Get the structure in writing before you sign.

• A good answerThe cover period for defects is stated in the contract, each change is quoted before work starts, and any ongoing plan is optional and explained in plain words.
• Red flagSupport terms appear only after launch, or a support plan must start before anything is live.

15. Is each phase scoped, priced and signed on its own?

Gartner's same release predicts that over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls. Phases signed one at a time give you a clean exit after each one.

• A good answerPhase one is one job with its own fixed price and a clear finish line. You decide on phase two after you have seen phase one work.
• Red flagOne contract for the whole programme, with most of the money due before anything is live.

Twelve AI agency red flags

These patterns recur across the rival guides, the Reddit threads the assistants cited and the regulators' pages we read on 18 October 2026. One is a reason to ask harder questions. Three is a reason to walk away.

1. Agent washing. A chatbot or scripted automation sold as an agent. Gartner puts the real agentic vendors at about 130 of thousands.
2. A 20-slide deck and no live agent. If they cannot show one running, assume there is not one.
3. "It learns on its own." Ask what exactly changes, who approves it and how it is tested. Most agents change only when someone edits the prompts, data or code.
4. Keys and servers only in their account. You cannot see usage, and leaving means a rebuild.
5. No named people. You meet a salesperson and never the builder.
6. A price only after a long paid discovery. You pay to learn whether you can afford the project.
7. No exit or handover clause. Nothing says what you receive if the work stops: code, prompts, test set, logs.
8. A support plan that starts before anything is live. Ongoing fees before there is anything to support.
9. A single accuracy number with no test set. "95% accurate" on what, measured how?
10. No answer on prompt injection. Or a reply that it is the model provider's problem.
11. An admin account for the agent. No list of actions, no approval step for the risky ones.
12. Claims the agent replaces a profession. The FTC said in its September 2024 Operation AI Comply release that there is no AI exemption from the laws on the books. Big claims need tests behind them.

Several of these line up with our guide to why AI agent projects fail: the failure usually starts in the contract, long before the first line of code.

A scorecard you can use on three AI companies

The Sparked Group's scorecard, published 16 October 2026, uses ten weighted criteria and treats a score of 1 on diagnosis, governance or ownership as a veto. Ours keeps the idea of weights and vetoes, and builds the rows from the 15 questions above, so prompt injection, approvals, the test set and data roles each get a score. Set the weights before you read any proposal. Score each firm 1, 3 or 5 per row on what you have seen, then multiply by the weight and divide by 5. The best possible total is 100.

Arithmetic, so you can check it: a firm scoring 5 on every row reaches 100. A firm scoring 3 on every row reaches 60. A firm scoring 5 everywhere except a 1 on ownership has 88 points, and still fails, because a 1 on a veto row ends the review whatever the total says. Our working rule is to take firms above 75 to a demo. That threshold is our own, not an industry standard.

How vetting changes with company size

Turning the questions into an AI vendor questionnaire

An AI vendor questionnaire is the written form of this list, sent before any call. Larger companies already run one as part of AI vendor due diligence. 1up's list of the top 20 questions(22 May 2025) shows what AI software sellers receive in deal after deal: legal, security and data questions. Lets-viz's six-gate checklist for small businesses runs from data flow mapping to exit rights and says the review takes two to three working days.

For a custom build, those templates miss the parts that decide whether you can ever leave. Add three lines to whichever AI vendor risk assessment you already use:

1. List every item we will own at the end of each phase: code, prompts, test set, logs, documentation.
2. Name the account that will hold each model key, server and database, and who can switch access off.
3. Describe the demo you will build on a sample of our data before we sign, and how you will delete that sample afterwards.

NIST AI 600-1 action GV-6.1-009 backs this up: it asks organisations to update due diligence for generative AI procurement to include intellectual property, data privacy, security and other risks. Our AI agent governance policy guide shows where the answers go once you have them.

Reference-call questions to ask a past client

Ask the vendor for two clients whose projects went live at least six months ago. Call them without the vendor on the line. Six questions:

1. What does the agent do today, and how many people use it or rely on it each week?
2. What broke in the first three months after launch, and how fast was it fixed?
3. Were the people who built it the same people who sold it to you?
4. Do you hold the code, prompts and logs in your own accounts today?
5. How were changes after launch priced, and were there any surprises on the invoice?
6. Would you hire them again for a second project, and what would you change in the contract?

Extra checks in the UK and Australia

For UK firms, our list of AI agent development companies in the UK and the UK AI agents page are a starting point for the long list. In Australia, start with our guide to AI agencies in Australia and the Australian AI agents page. US buyers can compare our lists of AI agent development companies and AI automation agencies in the USA. Whichever best AI agency list you start from, run every name through the same 15 questions.

Related decisions before you sign

• Still deciding who should build it? Read AI agency vs in-house vs freelancer.
• Ready for the paperwork? Use the AI agent development contract checklist.
• Weighing the security answers? See AI agent security risks and controls.
• Want the services side? See AI agent development, AI automation and AI consulting.

How FactoryJet answers these 15 questions

We build custom AI agents, AI voice agents, and integrations with Odoo, NetSuite, SAP Business One, ERPNext, Salesforce, HubSpot, Xero, WhatsApp and EDI. Here is where our standard terms answer the questions above. Bring the rest to the scoping call and score us like anyone else.

Sources

Every page below was opened and read on 18 October 2026.

• NIST, AI Risk Management Framework (released 26 January 2023, being revised)
• NIST AI 600-1, Generative AI Profile (July 2024): GV-6.1-004, GV-6.1-008, GV-6.1-009, GV-6.2-007
• OWASP Gen AI Security Project, Top 10 for LLM Applications 2025, with LLM01, LLM03 and LLM06
• ICO, What are the accountability and governance implications of AI? (updated 15 March 2023)
• ICO, What needs to be included in the contract? (UK GDPR Article 28(3))
• OAIC, Guidance on privacy and the use of commercially available AI products (21 October 2024, updated 17 January 2025)
• UK Government, Guidelines for AI procurement (8 June 2020)
• Gartner, Over 40% of Agentic AI Projects Will Be Canceled by End of 2027 (25 June 2025)
• FTC, Crackdown on Deceptive AI Claims and Schemes (Operation AI Comply) (25 September 2024)
• Anthropic, Model deprecations
• ISO, ISO/IEC 42001:2023 AI management systems
• Buyer guides compared: BotsCrew, The Sparked Group, Dan Cumberland Labs, Telliant, 1up, Lets-viz, APSCO Australia
• Our own measurements: Google volumes and results for the US, UK and Australia (DataForSEO), and 36 buyer questions across five AI engines, all on 18 October 2026