AutoBench LLM Leaderboard

Overall Model Performance

Models ranked by AutoBench score. Lower cost ($ Cents), latency (s), and fail rate (%) are better. Iterations shows the number of evaluations per model.

Benchmark Correlations: AutoBench features 82.71% with Artificial Analysis Intelligence Index, 80.25% with Terminal-Bench Hard, 81.81% with GDPval-AA, 66.45% with Tau2-Bench Telecom.

Benchmark Comparison

This run targets agentic performance; reference scores are drawn from agentic benchmarks alongside the Artificial Analysis Intelligence Index.

Comparison of AutoBench scores with reference benchmarks: AAI Index, Terminal-bench, GDPval-AA, and Tau2-bench Telecom. Models sorted by AutoBench score (higher is better).

Benchmark Correlations: AutoBench features 82.71% with Artificial Analysis Intelligence Index, 80.25% with Terminal-Bench Hard, 81.81% with GDPval-AA, 66.45% with Tau2-Bench Telecom.

Performance Visualizations

Exploring relationships between AutoBench Rank, Latency, and Cost.

Rank vs. Average Cost

Rank vs. Average Latency

Rank vs. P99 Latency

Performance vs. Cost/Latency Trade-offs

Cost Breakdown per Domain ($ Cents/Response)

Average Latency Breakdown per Domain (Seconds)

P99 Latency Breakdown per Domain (Seconds)

Performance Across Different Domains

Model ranks within specific knowledge or task areas. Higher is better.

About AutoBench

AutoBench is an LLM benchmark where Large Language Models (LLMs) evaluate and rank the responses generated by other LLMs. The questions themselves are also generated by LLMs across a diverse set of domains and ranked for quality.

Methodology

1. Question Generation: High-quality questions across various domains (Coding, History, Science, etc.) are generated by selected LLMs.
2. Response Generation: The models being benchmarked generate answers to these questions.
3. Ranking: Ranking LLMs rank the responses from different models for each question, on a 1-5 scale.
4. Aggregation: Scores are averaged across multiple questions and domains to produce the final AutoBench rank.

Metrics

• AutoBench Score (AB): The average rank received by a model's responses across all questions/domains (higher is better).
• Avg Cost (USD Cents/response): Estimated average cost to generate one response based on model provider pricing (input+output tokens). Lower is better.
• Avg Latency (s): Average time taken by the model to generate a response. Lower is better.
• P99 Latency (s): The 99th percentile of response time, indicating worst-case latency. Lower is better.
• Chatbot Arena / Artificial Analysis Intelligence Index / MMLU: Scores from other well-known benchmarks for comparison (where available).

Data

This leaderboard reflects a run completed on April 23, 2025. Models included recently released models such as o4-mini, Gpt-4.1-mini, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet:thikning, etc..

Links

• AutoBench Run 2 Results
• AutoBench Blog Post
• Autobench Repositories

Disclaimer: Benchmark results provide one perspective on model capabilities. Performance can vary based on specific tasks, prompts, and API conditions. Costs are estimates and subject to change by providers. Latency depends on server load and geographic location.