Bring these 24 questions to discovery calls or proposal meetings. Ask each vendor for explanations and evidence before scoring. Leave unconfirmed items blank.
Production delivery
A working demo does not prove continued use. Check whether the vendor can handle production, acceptance and real operational problems.
- Which projects launched in the past year are still in use? With permission, can we speak to a user?
- Walk us through one move from testing to production. What were the acceptance criteria and your responsibilities?
- What failures or user issues occurred after launch? Who handled them, and how were fixes documented?
What a good answer looks like
They can explain production status, acceptance, team roles and ongoing responsibility within confidentiality limits, with verifiable records or an authorized reference.
Red flags
- Only videos or proofs of concept, with no clarity on actual users
- Claiming a partner’s delivery as their own without explaining their role
Data security and access
Establish which data AI touches and who can access it. Data exposure and permission failures should remain visible beyond an average score.
- List the model services and regions receiving our data, with retention, deletion and training-use settings.
- How are department and role permissions enforced? Can you demonstrate that an unauthorized account cannot retrieve content?
- Who manages credentials, access logs and incident reporting? Are response and notification responsibilities in the contract?
What a good answer looks like
A data-flow inventory, testable access controls and logs, clear provider terms and retention settings, and accountable incident handling.
Red flags
- Saying the model is secure without explaining data flows and retention
- Shared accounts or permissions for all users, with no unauthorized-access testing
Ownership and handover
You need to operate and maintain the work after the engagement ends. Agree rights, accounts and documentation before signing.
- What rights apply to source code, prompts, workflow configurations and documentation? Which third-party licenses restrict use?
- Who owns the cloud, model and database accounts? Can we obtain administrative access and billing information ourselves?
- Which deployment instructions, operating guides and test records are delivered? How will we verify our team can take over?
What a good answer looks like
A contractual deliverables list, usage rights and third-party restrictions. The customer controls key accounts and can verify handover by operating the system.
Red flags
- Withholding necessary code or configuration without explaining restrictions up front
- Vendor-only control of key accounts, with no export or migration path
Model and platform neutrality
Choices should fit your requirements, costs and constraints. The vendor should explain the selection, including any supplier relationships.
- Which viable models or platforms did you compare, and what data and metrics informed your choice?
- Do you receive commissions or have reseller or other commercial relationships with suppliers? How do you disclose them?
- If pricing, terms or quality change, which components can be replaced and what migration work is needed?
What a good answer looks like
Relevant alternatives, evaluation results and cost assumptions, disclosed supplier relationships, and clear replacement limits and migration work.
Red flags
- Claiming one model fits every requirement without testing
- Avoiding questions about commercial relationships or platform lock-in
Industry and workflow fit
The same model faces different exceptions and responsibilities in different workflows. The vendor needs to understand how the work gets done and where it fails.
- Which actual users will you interview, and how will you document our process and exceptions?
- Using a case we provide, show where AI would act and which steps would remain with people.
- Which industry terms, regulations or internal rules need confirmation? Who supplies and approves that knowledge?
What a good answer looks like
They investigate systems, users and exceptions, map a bounded workflow and acknowledge domain knowledge that requires customer confirmation.
Red flags
- Applying an existing demo before understanding the workflow
- Speaking only to managers, with no testing by actual users
Pricing and risk allocation
Compare quotes for equivalent scope and ongoing costs. Unclear changes, usage assumptions and acceptance terms invite disputes.
- Which deliverables and acceptance criteria are included, and which work, licenses and fees are excluded?
- How are model, cloud and maintenance costs calculated? List usage assumptions and charges above those levels.
- If scope changes or acceptance criteria are not met, how are work, payment and termination handled?
What a good answer looks like
An itemized scope, milestones and acceptance criteria, transparent recurring costs, and contractual terms for changes and unmet requirements.
Red flags
- A single total price with no scope or recurring costs
- Expecting acceptance without agreed criteria or a remedy for unmet requirements
Language and on-site support
Support should fit your language, working hours and collaboration needs. Decide whether on-site work is needed from the workflow itself.
- Which languages can you use for communication, documentation and training? Can you share an example?
- What time zones are the main contact and engineers in? Which meeting and support hours can they cover?
- Who can join on-site workflow discovery? How are remote and on-site responsibilities and fees agreed?
What a good answer looks like
Confirmed languages, time zones, named contacts and escalation paths, with on-site arrangements and fees based on actual requirements.
Red flags
- Promising support at any time without contacts or available hours
- Sales committing to on-site work without confirming engineering availability
Operations and expansion
Data, models and workflows change. Establish who monitors, fixes and extends the system after launch.
- Which quality, cost and error metrics will you monitor after launch? Who reviews them and handles anomalies?
- How are model or data-source updates regression-tested, versioned and rolled back?
- What maintenance is included? How are responsibilities, pricing and delivery agreed for new workflows or higher usage?
What a good answer looks like
A monitoring and regression-test plan, versioning and rollback procedures, and written maintenance scope, contacts and expansion terms.
Red flags
- No accountable operator or issue-reporting channel after delivery
- Pushing updates to production without testing or a rollback path