Evaluating AI agents: from a good impression to a useful result

Handover is the checkpoint

An agent receives a folder of documents to prepare a decision. Soon there is a neatly written summary. It looks finished. But were all relevant documents considered? Do the numbers match their sources? Are missing details still visible? And did the agent only prepare what it was supposed to prepare?

This is a constructed example, not a measured product trial. It illustrates a distinction I care about: the quality of the wording and the quality of the completed task are different questions.

For a meaningful test, I therefore start with the handover. I first describe what another person must be able to use at the end. This determines the agent’s evaluation criteria. “Was the answer good?” is too broad a question.

A small test brief

For this document example, the brief could be:

Area	Define before the trial
Task	Compare three offers against explicitly stated requirements.
Result	A comparison table with source references and open questions.
Limits	Do not fill in missing prices, contact suppliers or change files outside the working folder.
Failure case	Show conflicting information and leave the decision open.
Review	A person can trace every decision-relevant claim to its source.

This table is not a universal standard. It is a small aid adaptable to a concrete task. Preparing it can expose unclear expectations: should the system only compare, or also recommend? May it weigh sources? What information would prevent a recommendation?

Examine the outcome and process separately

First, check the final output: was the table created, are mandatory details included and do referenced files exist? Some of these questions can be answered with simple technical checks.

Then comes substantive review: is the comparison meaningful, was a limitation missed, and did an open question remain open? This needs someone with the necessary context or a carefully agreed evaluation standard.

The route matters too. A correct result might follow unauthorised access; a well-documented process can still produce a wrong result. Checking permissions therefore belongs alongside checking outcomes. Anthropic’s guide to agent evaluations likewise distinguishes the recorded process from the actual final state, and technical, model-based and human evaluations.

Include difficult cases from the start

For an initial small test set, I would not choose only complete, easy-to-read documents. A missing attachment, two conflicting versions and an inaccessible source may reveal more about handling uncertainty than another smooth run.

Include a case where the system should refuse or ask for clarification. If every test expects a complete solution, correctly stopping becomes invisible. Yet stopping may be decisive in real use.

Individual trials can differ. Repetition helps detect a lucky result. Document the task, inputs, system version, tool access and evaluation. A percentage without that information says little about what was actually tested.

Start small, decide concretely

The outcome is a bounded decision: which tasks is the workflow suitable for, where does it need extra checking, which permissions stay closed, and what must change before use in another setting?

This connection between workspace, workflow and handover is what interests me in agent development. Alpine-X is one of my project approaches. But a project name is not evidence. The next solid step is a traceable trial whose outcome someone else can review.