AutomationBench-AA evaluates agents working across simulated business applications. Its score measures the share of task objectives completed without guardrail violations. This objective-level metric is separate from the strict task-completion score on Zapier’s hosted AutomationBench leaderboard.
reported precision decimals: 1 · result quality: reported · Claude Opus 5.5 (max with fallback); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/claude-opus-5-5 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · GPT-6 Astra (max); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/gpt-6-astra · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Muse Wonder (max); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/muse-wonder · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · GPT-6.1 Sol (xhigh); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/gpt-6-1-sol-xhigh · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Grok 4.7 (xhigh); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/grok-4-7 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Gemini 3.8 Flash (medium); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/gemini-3-8-flash-medium · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · GPT-6.1 Terra (max); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/gpt-6-1-terra · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · GPT-5.6 Sol (max); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/gpt-5-6-sol · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · GPT-5.6 Terra (max); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/gpt-5-6-terra · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Claude Fable 5.1 (max with fallback); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/claude-fable-5-1 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · MiMo-V2.6-Pro; private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/mimo-v2-6-pro · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Muse Spark 1.3 (max); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/muse-spark-1-3 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Claude Opus 5 (max); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/claude-opus-5 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Claude Fable 5 (with fallback); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/claude-fable-5 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · GPT-6 Luna (max); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/gpt-6-luna · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Step 5 Preview; private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/step-5 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · GPT-5.6 Luna (max); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/gpt-5-6-luna · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Muse Spark Open (xhigh); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/muse-spark-open · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Qwen3.8 27B (xhigh); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/qwen3-8-27b · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · GPT-5.5 (xhigh); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/gpt-5-5 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Claude Opus 4.8 (max); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/claude-opus-4-8 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Muse Spark 1.2 (xhigh); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/muse-spark-1-2 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Muse Spark 1.1 (xhigh); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/muse-spark-1-1 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Claude Sonnet 5 (max); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/claude-sonnet-5 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: provisional · Claude Haiku 5.5 (xhigh); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. Publisher flags elevated refusals in pre-release testing and plans a rerun. Provisional result. · source model href: /models/claude-haiku-5-5-xhigh · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Gemini 3.1 Pro Preview; private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/gemini-3-1-pro-preview · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · GPT-5.4 mini (xhigh); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/gpt-5-4-mini · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · GLM-5.1; private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/glm-5-1 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Claude Sonnet 4.6 (max); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/claude-sonnet-4-6-adaptive · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Nemotron 3.5 Super VL; private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/nemotron-3-5-super-vl · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Claude 4.5 Sonnet; private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/claude-4-5-sonnet-thinking · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Qwen3.5 397B A17B; private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/qwen3-5-397b-a17b · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Inkling (xhigh); private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/inkling · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · MiniMax-M2.7; private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/minimax-m2-7 · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Qwen3.5 9B; private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/qwen3-5-9b · task count: 657 · Tools enabled · turn cap: 50
reported precision decimals: 1 · result quality: reported · Qwen3 14B; private held-out 657 tasks; 50 turns; objective completion with zero credit for any guardrail violation. · source model href: /models/qwen3-14b-instruct-reasoning · task count: 657 · Tools enabled · turn cap: 50
high
high
Independent evaluation evaluation setup
Kimi K3.1 Pro; Private 657 tasks; 50 turns; objective completion
MiniMax-M3.1-Flash; Private 657 tasks; 50 turns; objective completion
Independent evaluation evaluation setup
max with fallback
DeepSeek V4.1 Flash (Max); Private 657 tasks; 50 turns; objective completion
DeepSeek V4.1 Flash 1017 (High); Private 657 tasks; 50 turns; objective completion
DeepSeek V4.1 Flash 1017 (Max); Private 657 tasks; 50 turns; objective completion
GLM-5.3 (Max); Private 657 tasks; 50 turns; objective completion
GLM-5.3-Flash; Private 657 tasks; 50 turns; objective completion
Kimi K3 (Max); Private 657 tasks; 50 turns; objective completion
DeepSeek V4 Pro 0813 (Max); Private 657 tasks; 50 turns; objective completion
DeepSeek V4 Pro 0424 (Max); Private 657 tasks; 50 turns; objective completion
GLM-5.3 (Low); Private 657 tasks; 50 turns; objective completion
Kimi K3 (Low); Private 657 tasks; 50 turns; objective completion
DeepSeek V4.1 Flash 1017 (Non-reasoning); Private 657 tasks; 50 turns; objective completion
DeepSeek V4.1 Flash (Non-reasoning); Private 657 tasks; 50 turns; objective completion
DeepSeek V4 Pro 0813 (Non-reasoning); Private 657 tasks; 50 turns; objective completion
DeepSeek V4 Flash 0420 (High); Private 657 tasks; 50 turns; objective completion
GLM-5.2 (Max); Private 657 tasks; 50 turns; objective completion
DeepSeek V4 Flash 0420 (Max); Private 657 tasks; 50 turns; objective completion
MiniMax-M3; Private 657 tasks; 50 turns; objective completion
Kimi K2.6 (Reasoning); Private 657 tasks; 50 turns; objective completion
DeepSeek R1 (Jan '25); Private 657 tasks; 50 turns; objective completion
gpt-oss-120b (High); Private 657 tasks; 50 turns; objective completion
gpt-oss-20b (High); Private 657 tasks; 50 turns; objective completion
IFBench tests whether models follow precise output requirements and generalize to 58 diverse, verifiable constraint types.
GPQA DiamondGPQA Diamond tests graduate-level scientific knowledge and reasoning with difficult multiple-choice questions in biology, chemistry, and physics.
BrowseCompBrowseComp evaluates web research agents on questions with hard-to-find but verifiable answers.
