Reasoning
198-question Diamond subset; independent AA evaluation
- 01
- 02
- 03
- 03
- 03
- 29
DeepSeek R1 (Jan 2025)
Across the task benchmarks we track, DeepSeek R1 (Jan 2025)’s strongest relative results are in terminal use, work automations, and reasoning. On Terminal-Bench 2.1, it scores 19.1%, ranking #12 among 13 models with reported results in our dataset.
Updated: Data as of:
Strongest results
| Benchmark | Measures | Score |
|---|---|---|
| Terminal-Bench ↗4.0 / mini-swe-agent | Working in a terminal | 0%Joint #49 of 56 |
| Humanity's Last Exam ↗Stable HLE — text-only subset | Expert reasoning | 8.5%#64 of 68 |
| AutomationBench-AA ↗1.0.6 / AA private 657 | Work automations | 0.24%#52 of 55 |
| AA-Briefcase ↗1.1 / combined Elo | Business deliverables | 128.91 Elo#54 of 57 |
Model details
| Model | DeepSeek R1 (Jan 2025) |
|---|---|
| Agent Index v1.2 | Insufficient coverage (3/7) · Preview |
| Creator | DeepSeek (opens in a new tab) |
| Availability | Open source / open weights |
| Context size | 128,000 tokens (opens in a new tab) |
| Max output | Not reported |
| Knowledge cutoff | Not reported |
| License | MIT · Model weights ↗ (opens in a new tab) |
Pricing and performance
Per provider and reported configuration. Prices are USD per 1M tokens.
No provider pricing or performance reports have been added for this model yet.
Benchmark performance
DeepSeek R1 (Jan 2025) in color, alongside leading models. Latest stored reports, best effort per report date.
GPQA Diamond ↗
Coding
Verified split; published reports
- 01
- 02
- 03
- 04
- 05
- 07
Terminal Use
89 tasks; Terminus 2; e2b; three repeats
- 01
- 02
- 03
- 04
- 05
- 12
Work Automations
Private 657 tasks; 50 turns; objective completion
- 01
- 02
- 03
- 04
- 05
- 52
Knowledge Work
220 tasks; Stirrup agent; pairwise Elo
- 01
- 02
- 03
- 04
- 05
- 38
Reported benchmarks
All published results in our dataset, with versions, evaluation settings, and sources.
| Benchmark | Score | Evaluation setup | Source |
|---|---|---|---|
| AA-Briefcase1.1 / combined EloCombined Elo · Higher is better | 128.91 Elo | selection: best reported configuration per release · DeepSeek R1 (Jan '25) · tools: benchmark-defined | Artificial Analysis (opens in a new tab)independent |
| AutomationBench-AA1.0.6 / AA private 657Objectives completed without guardrail violations · Higher is better | 0.242% | DeepSeek R1 (Jan '25); Private 657 tasks; 50 turns; objective completion | Artificial Analysis (opens in a new tab)independent |
| GDP.pdf100 tasks / All-pass / five attemptsAccuracy · Higher is better | 2.4% | selection: best reported configuration per release · DeepSeek R1 (Jan '25) · tools: benchmark-defined | Artificial Analysis (opens in a new tab)independent |
| GDPval-AA2.1Elo · Higher is better | 302.47 Elo | DeepSeek R1 (Jan '25); 220 tasks; Stirrup agent; pairwise Elo | Artificial Analysis (opens in a new tab)independent |
| GPQADiamond / Artificial AnalysisAccuracy · Higher is better | 70.808% | DeepSeek R1 (Jan '25); 198-question Diamond subset; independent AA evaluation | Artificial Analysis (opens in a new tab)independent |
| Humanity's Last ExamStable HLE — text-only subsetAccuracy · Higher is better | 8.5% | DeepSeek R1 (Jan '25); original HLE text-only 2,158 questions; no tools; not HLE Diamond · No tools | Artificial Analysis (opens in a new tab)independent |
| IFBench58 constraint typesAccuracy · Higher is better | 38.98% | selection: best reported configuration per release · DeepSeek R1 (Jan '25) · tools: benchmark-defined | Artificial Analysis (opens in a new tab)independent |
| SWE-benchVerified / published open-model reportsResolved · Higher is better | 49.2% | harness comparison: provider-reported; settings differ across models · Original January 2025 R1 model card; SWE Verified resolved; not R1-0528 | DeepSeek ↗ (opens in a new tab)provider reported |
| Terminal-Bench2.1 / Artificial AnalysisPass@1 · Higher is better | 19.101% | DeepSeek R1 (Jan '25); 89 tasks; Terminus 2; e2b; three repeats | Artificial Analysis (opens in a new tab)independent |
| Terminal-Bench4.0 / mini-swe-agentTask success · Higher is better | 0% | harness: mini-swe-agent · repeats: 3 · selection: best reported configuration per release · DeepSeek R1 (Jan '25) | Artificial Analysis (opens in a new tab)independent |