Choosing an AI supplier requires evidence about the proposed system, the data it will use and the people who will rely on its output. A convincing demonstration can reveal useful capabilities, but it does not show how the service performs on your own tasks or how failures will be handled.
Test the system against the intended use
Describe the workflow before collecting vendor documents. Identify the inputs, the expected output and the decisions that require human approval. Ask which model and tools the service uses, whether they can change during the contract, and how the supplier communicates those changes.
Build an evaluation set from representative cases, including incomplete inputs and situations where the correct response is to ask for help. Compare outputs with an agreed reference. Record factual errors, unsupported citations and failures to follow instructions separately; an average score can hide a serious weakness in one task.
Review the operating arrangements
Request clear answers about data retention, access, subcontractors, incident reporting and deletion. Establish what the supplier can demonstrate and what remains a contractual commitment. Avoid uploading confidential evaluation material until the relevant handling arrangements have been approved.
Keep the evaluation results alongside the proposed service limits, pricing and support process. Approval should identify the permitted uses and the conditions for reassessment. A model change, a new data source or an expansion into a higher-impact workflow can justify a fresh review.