LLM Model Benchmarks

This document reports how frontier LLMs perform when they power an AI Colleague — executing real multi-step Agentic Operating Procedures (AOPs) and calling platform tools — so you can choose the right model and provider for your deployment.

Every number comes from the same golden dataset of scenarios, run through the AI Colleague agent stack with real tool execution, and graded by an independent LLM judge. Results are comparable across models.

How to use this guide

• Quick decision? Start with the Overall results leaderboard and Choosing a model.
• Latency-sensitive deployment? See Latency by provider — the serving provider changes latency as much as the model does.
• Matching a model to a kind of work? See Use-case types in the golden dataset — how each model handles document reconciliation vs provisioning vs batch loops, and happy paths vs edge, negative, and adversarial scenarios.
• Validating a model for a specific use case? Jump to the use-case breakdown.
• Understanding the numbers? See How to read the numbers and Methodology.

---

Models, providers, and effort

Each model was benchmarked at low reasoning effort. Accuracy figures come from the run on the model's primary serving provider (listed below); the same benchmarks were then re-run on an alternate provider to measure how serving infrastructure affects latency.

Higher reasoning-effort tiers will be added in a future benchmark round.

---

Overall results

Accuracy is the macro-averaged pass rate across all 6 AOPs and 7 tools. Latency is time-to-first-token (TTFT), averaged across the 6 AOP suites on the model's primary provider.

Choosing a model

Directional takeaways from this round:

• Highest accuracy on multi-step AOPs: Claude Opus 5.5 led the AOP average (100.0%) and was the only model to fully pass all six AOPs.
• Fastest first token: Claude Sonnet 5.5 on Anthropic had the lowest average TTFT (41.6s) and the fastest first token on four of the six AOPs, while still placing third overall (95.5%).
• Tools are commoditised, AOPs are not. All eight models scored 94%+ on individual tools; the spread only opens up on multi-step AOPs (81–100%). Choose your model based on the AOPs you plan to run.
• Provider choice is a real latency lever — see the next section before you finalise infrastructure.

---

Latency by provider

We ran the same benchmarks on the same model across different serving providers. For the GPT models we compared OpenAI with Azure OpenAI; for the Claude family, Anthropic with AWS Bedrock. Figures are average TTFT across the 6 AOP suites; each is a single run per provider, so read the gaps as directional.

Two patterns worth noting:

• GPT-5.5 was consistently faster on OpenAI than on Azure OpenAI in this round — 58.9s vs 85.6s average TTFT, with Azure slower on every AOP. The newer GPT models were close across providers: GPT-6.1 Sol was about 2.5s faster on Azure OpenAI, while GPT-6 Astra and GPT-5.6 Terra were within 1s.
• The Claude family split: Opus 5.5 (53.9s vs 58.6s), Sonnet 5.5 (41.6s vs 43.7s), and Sonnet 4.6 (50.4s vs 64.9s) were faster on Anthropic, while Opus 4.7 was significantly faster on AWS Bedrock (49.0s vs 75.9s).

TTFT per AOP and provider

---

Use-case types in the golden dataset

The golden dataset is not a flat list of prompts — every scenario encodes a kind of work (the AOP's workload type) and a kind of test (the scenario type). Reading results along these two axes shows where models genuinely differ.

AOP workload types and complexity

The six AOPs fall into four workload types, ordered below from least to most complex. The mix column reads happy path · edge case · negative · adversarial.

Complexity reflects the structure of the workflow the agent must orchestrate — not how models scored on it:

• Moderate — a few tools in a linear, read-only flow (a fetch followed by analysis).
• High — several tools plus one demanding construct: multi-document reconciliation with tolerance rules, or a dozen query intents with role branching.
• Very high — many tools plus multiple constructs at once: approval gates, loops, generated code, and writes across several enterprise systems.

Structural complexity and model difficulty don't map one-to-one: the very-high-complexity onboarding and offboarding flows scored 78–100%, while the high-complexity LMS handler produced the round's lowest scores. Complexity tells you how much a run can go wrong; the pass rates tell you how often it did.

The dataset also mixes in multilingual triggers (Spanish and Portuguese variants of full flows), scripted multi-turn follow-ups (the test user answers the agent's questions mid-run), and file attachments (POs, invoices, GRNs) — so pass rates reflect complete conversations, not single prompts.

Accuracy by scenario type

AOP scenarios only (57), micro-averaged across all six AOPs on each model's primary provider.

Two Sonnet 4.6 runs where the grading service returned no result are excluded from its denominators.

Three things stand out:

• Happy paths are the hardest. That sounds backwards until you see what "happy path" means here: the longest end-to-end runs with the heaviest rubrics (a full onboarding with provisioning across four systems, or all twelve LMS query intents). Models rarely stumble on guardrails; they stumble on completing long flows perfectly.
• Claude Opus 5.5 was flawless, and four more models were flawless outside happy paths. Opus 5.5 passed all 57 AOP scenarios. GPT-6.1 Sol, Claude Sonnet 5.5, GPT-6 Astra, and GPT-5.5 passed all 30 edge, negative, and adversarial scenarios — every one of their misses was a full end-to-end run.
• Guardrail behaviour is strong across the board. No model skipped an approval gate on request or acted on a planted false premise in provisioning flows. The few negative-type misses were failures to stop after a rejected ALL-approve gate (Claude Sonnet 4.6 and Claude Opus 4.7, in Employee Offboarding).

Tool categories

The seven tools group into four categories. Pass rates are micro-averaged over the category's scenarios.

At the tool level the models are near-interchangeable: identical on document work and within one miss of each other on retrieval and computation. Most of the separation is in transactional writes, where Self Service Tasks (conversational workflow submissions) ranged from 70% to 90%.

---

Results by use case — AOPs

Pass rate is the share of scenarios whose run satisfied the scenario's rubrics. TTFT and P95 total response are measured on the model's primary provider. All runs at low reasoning effort.

Invoice Query Assistant

Workload type: data query & analysis · Complexity: moderate. Fetches user invoices and uses data analysis to answer queries about invoice data. 8 scenarios — 5 happy path, 2 edge case, 1 adversarial. Tools used: Invoice Fetch API, Data Analysis.

Notable pattern: the "total amount due this year" scenarios — which require distinguishing issue date from due date — caused most failures; Claude Opus 5.5 was the only model to pass every scenario. GPT-6.1 Sol, GPT-6 Astra, and GPT-5.6 Terra reported the AED total without stating that it was a draft invoice (Sol also miscounted the invoices), Claude Sonnet 5.5 failed the Portuguese variant by adding an unpaid-only filter, Sonnet 4.6 failed both variants, and Opus 4.7 failed one. All models resisted the adversarial scenario where the user's claimed count contradicts the data.

3-Way Matching Purchase Order Reconciliation

Workload type: document-centric reconciliation · Complexity: high. Reconciles a Purchase Order, Vendor Invoice, and Goods Receipt Note via 3-way matching, flags discrepancies in an exception table, and routes major discrepancies for approval. 11 scenarios — 2 happy path, 6 edge case, 2 negative, 1 adversarial. Tools used: Document Parsing, Request Approval, Send Notification.

Notable pattern: the strongest AOP in the benchmark — seven of eight models passed every scenario, including tolerance edge cases, currency mismatches, and the "missing GRN — must not fabricate a match" negative. GPT-5.6 Terra's one miss was the multiple-price-variance edge case: its exception table left out the freight overcharge ($85 on the PO vs $95 on the invoice), and it requested approval a second time after approval had already been granted.

New Hire Onboarding and System Provisioning

Workload type: approval-gated provisioning · Complexity: very high. Automates end-to-end system provisioning for new hires (Azure AD, Jira, Salesforce, MS Teams, groups) with manager approval gates before provisioning begins. 7 scenarios — 2 happy path, 3 edge case, 1 negative, 1 adversarial. Tools used: Employee Directory Search, Request Approval, Create User in Azure AD / Jira / Salesforce, Add User to Group, MS Teams access, Send Notification.

Notable pattern: the only failures were on the full English happy path with the any-approve gate — failed by Claude Sonnet 4.6 and Claude Opus 4.7; every other model passed it. All models correctly stopped when the approver rejected, asked instead of fabricating a missing manager, and refused to skip approval gates on the adversarial prompt.

Employee Offboarding

Workload type: approval-gated provisioning · Complexity: very high. Handles IT offboarding end-to-end: confirms offboarding, raises asset deprovisioning tickets, routes the access-revocation plan through an ALL-approve gate, revokes Jira/Azure AD access, removes AD/DL group memberships, and produces a termination summary. 9 scenarios — 2 happy path, 5 edge case, 1 negative, 1 adversarial. Tools used include Asset Deprovision, Request Approval, Get User Jira / Delete User Jira, Azure AD Deactivate, Remove User From Groups, For Each Loop, Document Parsing.

Notable pattern: the hardest scenario was the negative one — "approver rejects the ALL gate; workflow must stop with no revocation" — failed by Claude Sonnet 4.6 and Claude Opus 4.7; every other model passed it. GPT-5.6 Terra's two misses were both about reporting, not actions: on the "no approval response" edge case it didn't tell the user that revocation had auto-proceeded, and on the Spanish both-approve run it didn't name both approvers or both approval references. All eight models refused the adversarial request to skip the approval gate.

LMS Learning Query Handler

Workload type: data query & analysis · Complexity: high. Handles employee and manager queries about learning programs, courses, completions, and overdue training — 12 use cases across self-service and manager perspectives. 13 scenarios — 12 happy path, 1 adversarial. Tools used: LMS, Data Analysis, JavaScript Code Executor, For Each Loop, Send Notification.

Notable pattern: the toughest AOP in the round; Claude Opus 5.5 was the only model to pass every scenario. Three manager/self scenarios — "Manager — remind associates", "Manager — team started learning", and "Self — learning started" — failed on GPT-5.5, Claude Sonnet 4.6, and Claude Opus 4.7. "Manager — team started learning" also failed on GPT-6.1 Sol and GPT-5.6 Terra (both reported 35 records against the 72 expected), and GPT-6 Astra and Claude Sonnet 5.5 each missed "Manager — essential assigned to team". Straightforward self-service queries were near-universal passes.

MVR Check

Workload type: batch loop processing · Complexity: very high. Runs Motor Vehicle Record checks for eligible recruitment candidates: fetches candidates, computes eligibility with generated code, branches contact logic between SMS and email, and updates attempt counters in a loop. 9 scenarios — 4 happy path, 4 edge case, 1 negative. Tools used: Fetch User Details, Fetch Form Link, JavaScript Code Executor, Reminder, Update Attempts, For Each Loop.

Notable pattern: loop resilience separates the field — "candidate record missing a Job Application ID mid-loop" failed on Claude Sonnet 4.6 and Claude Opus 4.7; every other model handled it. Opus 4.7 also acted on a read-only status query that should not have triggered sends, and GPT-5.6 Terra's one miss was the same query: it correctly sent nothing, but listed a candidate who had used up the reminder limit as an ordinary pending candidate. All eight refused the consent-override request.

---

Results by use case — Tools

Beyond multi-step AOPs, each individual tool is exercised in isolation and graded the same way. Latency for tools is average TTFT on the model's primary provider; effort is low throughout.

Tool latency (Avg TTFT)

What each tool does

• Case Management — creates, views, and manages support cases/tickets, including on behalf of another employee, filling case forms conversationally.
• Data Analysis — filters, aggregates, groups, and summarises structured data (CSV/Excel/JSON) fetched earlier in the run.
• Document Parsing — extracts text, key fields, and tables from uploaded documents (PDFs, images).
• Employee Directory — looks up employee records to answer people queries and resolve identities (manager, department, designation, contact).
• JavaScript Code Executor — generates and executes JavaScript for calculations, data transformations, string/regex processing, and date math.
• Orders & Products List View — fetches tabular data from a workflow-backed list view with exact-match filters and free-text search.
• Self Service Tasks — runs a workflow conversationally (Leave Request, Payslip Retrieval, Employee Termination, Employee Information Update, Daily Check-In), collecting required fields turn-by-turn and submitting without a separate confirmation step.

---

How to read the numbers

Each scenario is graded by an independent LLM judge that returns pass/fail on a set of criteria. The metrics reported are:

Grading criteria

---

Methodology

• Golden dataset. 118 curated scenarios — 57 across 6 AOPs and 61 across 7 tools — each defining the user input, expected tool calls, expected behaviour, and explicit pass criteria. The mix deliberately includes happy paths, edge cases, negatives (the agent must refuse or stop), and adversarial prompts (the user tries to bypass controls or plant false premises).
• Real tool execution. Tools execute for real against the same benchmark data for every model, so every model sees identical inputs and results are comparable.
• Independent LLM judge. Every run is graded by an external LLM judge against the scenario's rubrics; scenarios where the grading service returned no result are excluded from the pass-rate denominator (a harness failure, not a model failure).
• Reasoning effort. All models ran at low reasoning effort. Higher effort tiers are planned for a future round.
• Latency measurement. TTFT and total response time are measured across the agent's full turn in a production-equivalent environment, not isolated API calls — they include orchestration, tool execution, and multi-call agent loops, which is why figures are higher than raw model TTFT.
• Single-run caveat. Each model × provider × use-case cell is one run over a small scenario set. Treat small deltas as noise; the results are directional.
• Model coverage. This page reports the models currently selectable on the platform. The benchmark round also covered additional frontier models under evaluation; full results for every model are in the open benchmark repository.
• Benchmark date. July–October 2026. Model providers update capabilities continuously; this page is re-run and updated periodically.

Per-scenario results, full scenario definitions, pass criteria, and the exact AOP instructions are published in the open benchmark repository: aic-benchmarks.

---

Glossary