Factories > Management & observability

Benchmarking factory agent configurations

Benchmarks compare model and runner configurations for a factory agent on a fixed set of tasks. Use the results to decide which configuration to use in production.

Benchmarks compare model and runner configurations for one factory agent on the same fixed tasks. Use a benchmark to test a change on representative work before you apply it to your factory.

How benchmarks work

A benchmark suite is a reusable collection of tasks that evaluates one factory agent. Each task has a prompt and “Correctness criteria,” which tell the built-in Correctness Scorer what a successful trial must do. A trial is one run of one task under one configuration. Repetitions create additional trials.

When you launch a suite, you choose the configurations to compare, the Scorers to use, and the number of repetitions. Warp runs every task once for every configuration and repetition, then scores the completed trials. The run keeps those inputs, so later changes to the suite do not change its past results.

Use a suite for one focused question, such as whether a less expensive model preserves the quality of a review agent’s work. Test a different agent or job with a separate suite.

Create and run a benchmark

To use Benchmarks, you need a factory with an agent to evaluate. Use a completed run from that agent when you want a task to reproduce real work, or write a task yourself.

In the Warp Factories web app, open your factory.
In the factory sidebar, click Benchmarks.
Click New benchmark.
Enter a name and optional description, then choose the agent to evaluate. The suite runs every task as that agent.
Under Tasks, click Add task. Select a completed run, then click Add task to create a task from it. To write a task instead, click Start from scratch instead.
Enter a task title, prompt, and “Correctness criteria.” The criteria are required and are the authoritative requirements the built-in Correctness Scorer uses for every trial. Add a pinned repository only when the task must begin from a specific repository commit.
Save the suite, then click Run. In the launch dialog, choose the model and runner for each configuration. Mark one configuration as the baseline when you want the results table to show each candidate’s difference from it.
Add configurations to compare, choose the Scorers that apply to the agent, and set “Repetitions.” The dialog shows the number of trials that the selected tasks, configurations, and repetitions create. More trials and Scorers increase the run’s cost.
Click Run benchmark. While it runs, the benchmark page shows completed, running, pending, failed, and scored trials. You can cancel a running or scoring benchmark from this page.

Review benchmark results

After the run completes, review the result as a comparison, not as a universal model ranking:

Overall recommendation - Identifies the highest-quality configuration when at least two configurations have comparable results.
Additional recommendations - Highlight the most efficient and lowest-cost configurations when the result supports those comparisons.
Comparison chart - Compare the selected result dimensions across configurations.
Overall table - Compare each Scorer’s average and the combined Overall value. Expand a configuration, task, and repetition to inspect its individual trials.
Scorer grids - Show each task’s results across configurations for a selected Scorer.

The run’s “Total cost” includes trial and Scorer costs. A failed or cancelled benchmark shows only results that finished scoring before the run stopped.

Apply a result

Change one configuration at a time. If the evidence supports a candidate, update the agent’s model or runner in the factory dashboard, or submit the change through your factory definition. Keep the relevant Scorers active, then compare later production runs with the baseline you recorded before the change.

For version-controlled factories, define reusable suites in benchmarks/<suite-slug>/suite.yaml and their tasks in benchmarks/<suite-slug>/tasks/<task-slug>.yaml. See benchmark suite files.

Measure and improve a factory - Configure Scorers and use benchmark evidence in an improvement loop.
Factory dashboard - Track factory work, runs, and benchmark suites.
Factory definition syntax - Define factories and benchmark suites as code.