Security questions to ask an AI automation vendor
We are a vendor in this category, so read this with that in mind. It is also the checklist we would want a client to hold us to, and we have included the questions we find hardest.
What is the single most revealing question?
Ask them to draw the data flow. A vendor who has genuinely designed for security can sketch, without preparation, what leaves your systems, which third parties receive it, and where it is retained. One who responds with reassurance instead of a diagram has not thought about it, and the rest of their proposal should be read in that light.
Questions about data
The goal is to establish what physically moves, not to collect promises. A vendor should be able to answer each of these in a sentence.
- What data leaves our systems, field by field? "Only what is necessary" is not an answer. A list is.
- Which processors receive it? Model provider, automation platform, and anything in between, named.
- Which API tier is used? Training exclusion is usually a property of the tier, not of a promise.
- How long is anything retained, and where? Including logs, which people forget are copies of your data.
- Can the task run on de-identified or partial data? Often it can, and a good vendor will have asked already.
Questions about permissions
This is where the difference between a careful build and a fast one shows most clearly, and it is the section buyers most often skip.
- What exactly can this write to? Every action, enumerated, not "it integrates with our CRM".
- What credentials does it hold, and at what scope? One all-powerful service account is a finding, not a design.
- Which actions require human approval? And who decided that list.
- How are credentials rotated? If the answer is that they are not, that is your answer.
Questions about failure
Everything works in the demo. What matters is the behaviour on the day it does not, and this is the area where inexperienced vendors are most easily identified.
- What happens when a step fails halfway? A vendor who has not considered partial completion has not run anything in production.
- How do we find out something broke? Silent failure is the default unless somebody designed against it.
- What is logged, and could we reconstruct a decision six months later? Especially if you are regulated.
| Question | Good answer | Warning sign |
|---|---|---|
| What data leaves our systems? | A specific field list, offered in writing | "Everything is encrypted" — which does not answer the question |
| Is our data used for training? | Names the tier and the contractual term excluding it | "No" with nothing behind it |
| What can it write to? | An enumerated list of actions with scopes | "It has access to your CRM" |
| What needs approval? | A named list plus who signed it off | "It only does safe things" |
| What happens on failure? | Held item, alert, retry path, owner | "It retries automatically" and nothing more |
| Who owns the result? | You do; built in your accounts, documented | Deliverable runs on the vendor's platform only |
Should you demand certifications?
Match the requirement to the actual risk. A formal attestation is reasonable when a vendor hosts your data on their infrastructure. It means considerably less when they build inside your own accounts and hold nothing themselves — and reflexively demanding it filters out capable small firms while telling you little about the ones that remain.
If you do ask, ask what the certification covers. Scope varies enormously between two vendors claiming the same badge. A shared vocabulary helps here: the NIST AI Risk Management Framework gives both sides language for data handling, human oversight and failure detection that does not depend on anyone's marketing.
What should end the conversation?
Any variant of "trust us". Concretely: not knowing which model provider the build uses, being unable to enumerate what an agent can write to, having no answer on partial failure, or resisting putting the data flow in writing. None of these are hard questions for someone who has built this before, which is exactly why the inability to answer them is informative.
Related reading: the difference between an agent and a chatbot explains why the permission questions matter far more for one than the other, and what automation costs covers why governance rather than intelligence drives the price. These questions are only half of the diligence — the commercial half, covering account ownership, handover and scope discipline, is in how to choose an AI automation consultant. Our own position on all of this is set out on the about page.
Questions buyers ask us about this list
What is the single most revealing question to ask?
Draw me the data flow. A vendor who has genuinely thought about security can sketch what leaves your systems, which processors receive it, and where it is retained, without preparation. One who answers with reassurance rather than a diagram has not designed for it, and everything else in their proposal should be read in that light.
Is a signed agreement enough on training exclusion?
It is necessary but not sufficient. Ask which API tier the build actually uses, because training exclusion is usually a property of the tier rather than of a promise. A vendor building on a consumer plan while assuring you of enterprise terms is describing a contract that does not match the architecture.
Should we require a specific certification?
Match the requirement to the risk rather than demanding certifications reflexively. A formal attestation is reasonable when a vendor hosts your data; it is less meaningful when they build inside your own accounts and hold nothing. Ask what the certification actually covers, because scope varies enormously between two vendors claiming the same one.
What answer should end the conversation?
Any version of trust us. Specifically: not knowing which model provider is used, being unable to enumerate what an agent can write to, having no answer on what happens when a step fails, or resisting a written data-flow description. None of these are difficult questions for a vendor who has built this before.
Free consultation
Tell us what is costing you the most time
Describe the process. You get a straight read on whether it is worth automating, roughly what it would take, and what we would tackle first — on a call that costs nothing.
- Under an hour, free, and no obligation
- Fixed price per stage, agreed before any work starts
- You own everything we build, documented and in your accounts
- If you do not need us, we will say so on that call
The form is the fastest way to reach us — we reply from a real person, usually within one business day.