> For the complete documentation index, see [llms.txt](/llms.txt). > Markdown versions of each page are available by appending .md to any URL. # Benchmarking factory agent configurations Benchmarks compare model and runner configurations for a factory agent on a fixed set of tasks. Use the results to decide which configuration to use in production. Note Warp Factories is in **Early Access** and available to a limited set of teams. [Request access](https://www.warp.dev/factories/request-access) to use it with your team. Benchmarks compare model and runner configurations for one factory agent on the same fixed tasks. Use a benchmark to test a change on representative work before you apply it to your factory. ## How benchmarks work A benchmark suite is a reusable collection of tasks that evaluates one factory agent. Each task has a prompt and “Correctness criteria,” which tell the built-in Correctness Scorer what a successful trial must do. A trial is one run of one task under one configuration. Repetitions create additional trials. When you launch a suite, you choose the configurations to compare, the Scorers to use, and the number of repetitions. Warp runs every task once for every configuration and repetition, then scores the completed trials. The run keeps those inputs, so later changes to the suite do not change its past results. Use a suite for one focused question, such as whether a less expensive model preserves the quality of a review agent’s work. Test a different agent or job with a separate suite. ## Create and run a benchmark To use Benchmarks, you need a factory with an agent to evaluate. Use a completed run from that agent when you want a task to reproduce real work, or write a task yourself. 1. In the [Warp Factories web app](https://platform.warp.dev), open your factory. 2. In the factory sidebar, click **Benchmarks**. 3. Click **New benchmark**. 4. Enter a name and optional description, then choose the agent to evaluate. The suite runs every task as that agent. 5. Under **Tasks**, click **Add task**. Select a completed run, then click **Add task** to create a task from it. To write a task instead, click **Start from scratch instead**. 6. Enter a task title, prompt, and “Correctness criteria.” The criteria are required and are the authoritative requirements the built-in Correctness Scorer uses for every trial. Add a pinned repository only when the task must begin from a specific repository commit. 7. Save the suite, then click **Run**. In the launch dialog, choose the model and runner for each configuration. Mark one configuration as the baseline when you want the results table to show each candidate’s difference from it. 8. Add configurations to compare, choose the Scorers that apply to the agent, and set “Repetitions.” The dialog shows the number of trials that the selected tasks, configurations, and repetitions create. More trials and Scorers increase the run’s cost. 9. Click **Run benchmark**. While it runs, the benchmark page shows completed, running, pending, failed, and scored trials. You can cancel a running or scoring benchmark from this page. Caution Third-party harness comparison is not available in the current launch. The harness menu marks those choices as coming soon. ## Review benchmark results After the run completes, review the result as a comparison, not as a universal model ranking: - **Overall recommendation** - Identifies the highest-quality configuration when at least two configurations have comparable results. - **Additional recommendations** - Highlight the most efficient and lowest-cost configurations when the result supports those comparisons. - **Comparison chart** - Compare the selected result dimensions across configurations. - **Overall table** - Compare each Scorer’s average and the combined Overall value. Expand a configuration, task, and repetition to inspect its individual trials. - **Scorer grids** - Show each task’s results across configurations for a selected Scorer. The run’s “Total cost” includes trial and Scorer costs. A failed or cancelled benchmark shows only results that finished scoring before the run stopped. ## Apply a result Change one configuration at a time. If the evidence supports a candidate, update the agent’s model or runner in the factory dashboard, or submit the change through your [factory definition](/factories/factory-as-code/). Keep the relevant Scorers active, then compare later production runs with the baseline you recorded before the change. For version-controlled factories, define reusable suites in `benchmarks//suite.yaml` and their tasks in `benchmarks//tasks/.yaml`. See [benchmark suite files](/factories/factory-as-code/#benchmarkssuite-slugsuiteyaml). ## Related pages - [Measure and improve a factory](/factories/measure-and-improve/) - Configure Scorers and use benchmark evidence in an improvement loop. - [Factory dashboard](/factories/factory-dashboard/) - Track factory work, runs, and benchmark suites. - [Factory definition syntax](/factories/factory-as-code/) - Define factories and benchmark suites as code.