AutoBench LLM Leaderboard
Interactive leaderboard for AutoBench, where LLMs rank LLMs' responses. Includes performance, cost, and latency metrics. Use the dropdown below to navigate between different benchmark runs.
Choose a benchmark run to view its results
Current Run: AutoBench Run 3 - August 2025 (2025-08-14) - 34 models
Overall Model Performance
Models ranked by AutoBench score. Lower cost ($ Cents), latency (s), and fail rate (%) are better. Iterations shows the number of evaluations per model.
Benchmark Correlations: AutoBench features 86.85% with LMArena, 92.17% with Artificial Analysis Intelligence Index, 75.44% with MMLU.
Overall Rankings
llama-3_1-Nemotron-Ultra-253B-v1 | 4.511567341 | 4.368 | 89.99818067 | 277.6722 | 19.27% | 385 |
Benchmark Comparison
Comparison of AutoBench scores with other popular benchmarks. AutoBench features 82.51% correlation with Chatbot Arena, 83.74% with Artificial Analysis Intelligence Index, and 71.51% with MMLU. Models sorted by AutoBench score.
Benchmark Comparison
llama-3_1-Nemotron-Ultra-253B-v1 | 4.511567341 | 1481 | 68950 | 0.871 |
Performance Visualizations
Exploring relationships between AutoBench Rank, Latency, and Cost.
Rank vs. Average Cost
Rank vs. Average Latency
Rank vs. P99 Latency
Performance vs. Cost/Latency Trade-offs
Cost Breakdown per Domain ($ Cents/Response)
Cost Breakdown
llama-3_1-Nemotron-Ultra-253B-v1 | 18.544 | 0.596 | 0.783 | 0.688 | 0.701 | 0.829 | 7.756 | 0.865 | 0.762 | 0.771 | 0.826 |
Average Latency Breakdown per Domain (Seconds)
Average Latency Breakdown
llama-3_1-Nemotron-Ultra-253B-v1 | 238.2024 | 31.7501 | 13.1592 | 11.2295 | 18.9687 | 13.7099 | 302.2976 | 271.5234 | 10.8395 | 11.2754 | 11.51902452 |
P99 Latency Breakdown per Domain (Seconds)
P99 Latency Breakdown
llama-3_1-Nemotron-Ultra-253B-v1 | 336.5838 | 127.0559 | 115.4247 | 155.9084 | 183.8738 | 122.7531 | 934.9063 | 354.6674 | 147.6845 | 173.4233 | 246.0464 |
Performance Across Different Domains
Model ranks within specific knowledge or task areas. Higher is better.
Domain Performance
llama-3_1-Nemotron-Ultra-253B-v1 | 3.4733 | 3.8572 | 3.7443 | 4.4302 | 3.7371 | 3.8701 | 2.8162 | 2.7809 | 4.4164 | 3.8134 | 3.586292962 |
About AutoBench
AutoBench is an LLM benchmark where Large Language Models (LLMs) evaluate and rank the responses generated by other LLMs. The questions themselves are also generated by LLMs across a diverse set of domains and ranked for quality.
Methodology
- Question Generation: High-quality questions across various domains (Coding, History, Science, etc.) are generated by selected LLMs.
- Response Generation: The models being benchmarked generate answers to these questions.
- Ranking: Ranking LLMs rank the responses from different models for each question, on a 1-5 scale.
- Aggregation: Scores are averaged across multiple questions and domains to produce the final AutoBench rank.
Metrics
- AutoBench Score (AB): The average rank received by a model's responses across all questions/domains (higher is better).
- Avg Cost (USD Cents/response): Estimated average cost to generate one response based on model provider pricing (input+output tokens). Lower is better.
- Avg Latency (s): Average time taken by the model to generate a response. Lower is better.
- P99 Latency (s): The 99th percentile of response time, indicating worst-case latency. Lower is better.
- Chatbot Arena / Artificial Analysis Intelligence Index / MMLU: Scores from other well-known benchmarks for comparison (where available).
Data
This leaderboard reflects a run completed on April 23, 2025. Models included recently released models such as o4-mini, Gpt-4.1-mini, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet:thikning, etc..
Links
Disclaimer: Benchmark results provide one perspective on model capabilities. Performance can vary based on specific tasks, prompts, and API conditions. Costs are estimates and subject to change by providers. Latency depends on server load and geographic location.