The assistant answers quickly, speaks Arabic and displays an order number. It looks convincing, but the business question remains: did the number come from a real test system, does it belong to this user, and did the action happen only once? When selecting an AI-agent development company, request evidence of the outcome as well as fluent conversation.
An introductory meeting does not need to become a full technical audit. You need a limited demonstration that reveals the implementer's reasoning and the boundaries of what is shown, followed by agreement on broader testing before operation. A clear task makes it easier to distinguish a useful solution from a model showcasing attractive answers.
Agree on one observable task
Consider a fictional office-supplies distributor. Its assistant checks order status and records an address-change request for employee confirmation. It reads permitted information, verifies the customer through an agreed identity process and routes the change for review. It is not assumed to have permission to alter every order or approve every exception.
Define the task's beginning and end: who asks, which information is allowed, where the change request appears and who reviews it. State what is outside the demonstration, such as taking payments or cancelling financial documents. “An agent for sales and customer service” is too broad to be an acceptance criterion on its own.
Separate what the agent says from what happened
Anthropic's agent-evaluation guidance distinguishes a trial's record from the final outcome in its environment. For a buyer, the practical interpretation is simple: if the assistant says a request was recorded, ask to see the test record and its status instead of accepting the success message alone.
Ask whether the demonstration uses fixed data, a simulated integration or an environment closer to production. Simulation is useful early on when disclosed, but it does not establish that real integration, access controls or failure handling are ready.
A small case set is better than one easy question
| Test case | What to observe |
|---|---|
| A customer asks about an authorized order | An answer grounded in the correct record without unnecessary data exposure. |
| A question lacks approved information | Acknowledgment of the gap or a handoff, without an invented answer. |
| A request for another customer's data | Unauthorized access is denied while the original task remains intact. |
| A change requiring approval | A review request is recorded instead of executing the change directly. |
| An ambiguous or incomplete message | A useful clarifying question without a risky assumption. |
| A connected system fails | No false success claim, with a clear follow-up route. |
| A repeated request | Behavior follows a declared rule preventing unintended duplicate action. |
| The customer requests an employee | A handoff transfers appropriate context within access boundaries. |
This is a proposed starting point, not certification based on eight questions. Add cases from your operation, including typing errors, similar names and changed information. Use fictional, de-identified or otherwise authorized data. Do not send customer files merely to try a sales demonstration.
Define success before seeing the answer
For each case, record the input, what should happen, what must not happen and the evidence required. For example: exactly one address-review request is associated with the correct customer; the shipment address remains unchanged before approval; the reviewer can see the reason. This is clearer than “the bot understands customers.”
Assess factual accuracy, task completion, data protection and handoff separately. A good average does not cancel a case exposing another customer's information. Agree on failures that prevent launch regardless of other convincing responses, and identify who reviews results and approves exceptions.
Do not judge the system from one attempt
Outputs can change between attempts or when a question is rephrased. Try equivalent cases in the language your customers use and record the test configuration and outcomes. OpenAI's evaluation guidance describes traces and test datasets for reviewing behavior and changes. As a buyer, you need evidence you can inspect later.
Do not ask the company to conceal unsuccessful attempts. Ask what caused them, what changed and how the correction was tested. Repeating only successful questions after a change can hide a new failure elsewhere. Keep a reference set covering both useful behavior and important boundaries.
Ask who manages knowledge and tools
Who approves product prices or return information used by the agent? How is outdated information withdrawn? Which system owns the current order status? If the answer lives only in a long instruction that your team cannot update, discuss the maintenance workflow before contracting.
For tools, request a business-language description of what the agent can read, create and change and what requires review. Text instructing it to be cautious is insufficient if its accounts permit wider actions than needed. Ask the implementer to demonstrate access boundaries without requiring unnecessary production privileges for the demonstration.
Turn the demo into a pilot scope
Request a short document identifying the task, channel, knowledge source, integrations, roles and acceptance cases, including dependencies on third-party approvals or accounts. Specify the follow-up owner, how an action can be stopped when something goes wrong, how issues are classified and which data will measure the result.
Separate setup costs from operation, usage, monitoring and updates. If the supplier proposes a savings percentage or sales increase, ask what supports it and how it would be measured in your business. A successful demonstration on fictional records is not an achieved financial outcome.
For the distinction between conversation and task execution, read the business AI-agents guide. When discussing AI-agent development, send your task and proposed case set. A useful meeting ends with a testable agreement about a valuable task and its boundaries rather than a promise that an agent can do everything.
References and comparison method
Anthropic's agent-evaluation guidance and OpenAI's agent-workflow evaluation guidance were checked on 3 November 2026. Outcome, trace and repeatable-testing concepts are summarized briefly; the scenario, table and procurement framework are original. This checklist guarantees neither security nor a commercial result. The content was prepared with AI assistance and review of its text and sources.
