AI coding routes tested on Faros’s own engineering work.
Every route is tested against the same anonymized task cohort drawn from work Faros engineers actually completed. Compare quality, cost, cache use, reasoning setting, and recorded runtime—then inspect the evidence behind every result.
Powered by Time Machine
The selected results view defines one comparable cohort.
Routes are ranked by mean task-specific quality. Cost, cache use, task wins, and runtime when recorded remain separate observed measures.
Mean quality across the same task cohort · select a route for details
| Rank | Route | Quality score | Task wins | Cost / task | Cache hit | Runtime |
|---|
Quality whiskers are marginal 95% confidence intervals for each route’s mean across the task cohort. They describe score uncertainty; they are not rank ranges or paired-significance tests. Green cells mark the best observed value in each column.
Every published route is evaluated on the same task cohort and scoring rules.
—
—
Set your current route and monthly task volume. Cost and quality stay separate.
Uses Faros Engineering Benchmark averages. It is not observed production savings or a customer-specific evaluation.
No composite score hides the choice. The frontier shows quality against observed cost; the scorecard shows the rubric dimensions behind quality.
Up is better. Left is cheaper. The chart area is divided into four equal quadrants.
Average rubric dimensions by route
A tie counts as half a win. Every comparison uses the exact same task cohort.
Uses the selected cohort. Route A and Route B are highlighted across the matrix.
Slice the same official cohort without recomputing or reinterpreting the source files.
Official quality scores and public-safe task evidence are served from the catalog.
| Task | Task record | Domain | Complexity | Best route | Quality | Spread |
|---|
What was held fixed, what was measured, and what this result can support—presented with the evidence used to validate the instrument.
Direct answers about the workload, ranking, evaluation, cost, and what this public leaderboard does—and does not—represent.
Faros Route Index is a public catalog that compares complete AI coding routes using the same set of real Faros engineering tasks. It opens on the Leaderboard when a reviewed set of compatible official runs is available; otherwise it opens on the latest dated official run. Every dated run remains preserved for its task evidence and release-specific cost analysis.
No. It is evidence for the Faros Engineering Benchmark. A different organization, repository mix, or task distribution may produce a different ranking.
A route is the complete configuration used to do the work: coding harness, model, provider, reasoning or thinking setting, and relevant runtime settings. Faros Route Index compares routes because changing the harness or reasoning setting can change the outcome even when the model is the same.
Each benchmark version contains a stable cohort of anonymized tasks sampled from real Faros engineering work across product domains, task intents, and complexity levels. Every route in a run receives that same cohort.
Each generated change is graded against a task-specific rubric by a blinded judge. The quality score is the mean percentage of rubric credit earned; it is not blended with cost or runtime.
Cost uses observed token usage with the provider pricing configured for the run. Runtime is the observed end-to-end duration when captured; a dash means the source did not record it. Cache hit is the share of eligible input served from cache; it is not itself a discount and does not determine total cost.
New models, harnesses, and providers can be tested against the stable cohort as they become relevant. Every official benchmark run is dated, preserved, and selectable so results do not silently change.
No. This public page does not accept customer repositories. It publishes the Faros benchmark. Customer-specific optimization—connecting engineering data, identifying opportunities, and validating policies—is a separate Faros product workflow.