AutoBench LLM Leaderboard
Interactive leaderboard for AutoBench, where LLMs rank LLMs' responses. Includes performance, cost, and latency metrics.Data updated on April 25, 2025.
More info for this benchmark run: AutoBench Run 2 Results. If you want to know more about AutoBench: AutoBench Release.
Overall Model Performance
Models ranked by AutoBench score. Lower cost ($ Cents) and latency (s) are better.
llama-4-Maverick-17B-128E-Instruct-FP8 | 4.57 | 0.793 | 36.57 | 223.47 |
Benchmark Comparison
Comparison of AutoBench scores with other popular benchmarks. AutoBench features 82.51% correlation with Chatbot Arena, 83.74% with Artificial Analysis Intelligence Index, and 71.51% with MMLU. Models sorted by AutoBench score.
llama-4-Maverick-17B-128E-Instruct-FP8 | 4.57 | null | 69830 | 0.832 |
Performance Visualizations
Exploring relationships between AutoBench Rank, Latency, and Cost.
Rank vs. Average Cost
Rank vs. Average Latency
Rank vs. P99 Latency
Performance vs. Cost/Latency Trade-offs
Cost Breakdown per Domain ($ Cents/Response)
llama-4-Maverick-17B-128E-Instruct-FP8 | 0.329 | 0.154 | 0.161 | 0.163 | 0.152 | 0.178 | 0.172 | 10.232 | 0.152 | 0.161 | 0.183 |
Average Latency Breakdown per Domain (Seconds)
llama-4-Maverick-17B-128E-Instruct-FP8 | 15.48 | 11.25 | 10.95 | 10.37 | 13.08 | 10.82 | 136.12 | 205.27 | 13.77 | 14.83 | 15.53 |
P99 Latency Breakdown per Domain (Seconds)
llama-4-Maverick-17B-128E-Instruct-FP8 | 122.38 | 557.13 | 19.06 | 123.61 | 195.61 | 160.87 | 137.09 | 145.15 | 136.28 | 318.33 | 223.47 |
Performance Across Different Domains
Model ranks within specific knowledge or task areas. Higher is better.
llama-4-Maverick-17B-128E-Instruct-FP8 | 3.85 | 4.27 | 4.07 | 4.15 | 4.05 | 4.11 | 4.41 | 3.98 | 4.04 | 3.44 | 3.99 |
About AutoBench
AutoBench is an LLM benchmark where Large Language Models (LLMs) evaluate and rank the responses generated by other LLMs. The questions themselves are also generated by LLMs across a diverse set of domains and ranked for quality.
Methodology
- Question Generation: High-quality questions across various domains (Coding, History, Science, etc.) are generated by selected LLMs.
- Response Generation: The models being benchmarked generate answers to these questions.
- Ranking: Ranking LLMs rank the responses from different models for each question, on a 1-5 scale.
- Aggregation: Scores are averaged across multiple questions and domains to produce the final AutoBench rank.
Metrics
- AutoBench Score (AB): The average rank received by a model's responses across all questions/domains (higher is better).
- Avg Cost (USD Cents/response): Estimated average cost to generate one response based on model provider pricing (input+output tokens). Lower is better.
- Avg Latency (s): Average time taken by the model to generate a response. Lower is better.
- P99 Latency (s): The 99th percentile of response time, indicating worst-case latency. Lower is better.
- Chatbot Arena / Artificial Analysis Intelligence Index / MMLU: Scores from other well-known benchmarks for comparison (where available).
Data
This leaderboard reflects a run completed on April 23, 2025. Models included recently released models such as o4-mini, Gpt-4.1-mini, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet:thikning, etc..
Links
Disclaimer: Benchmark results provide one perspective on model capabilities. Performance can vary based on specific tasks, prompts, and API conditions. Costs are estimates and subject to change by providers. Latency depends on server load and geographic location.