LLM Model Benchmarks

This document reports how frontier LLMs perform when they power an AI Colleague — executing real multi-step Agentic Operating Procedures (AOPs) and calling platform tools — so you can choose the right model and provider for your deployment.

Every number comes from the same golden dataset of scenarios, run through the AI Colleague agent stack with deterministic, mocked tool outputs, and graded by an independent LLM judge. Results are reproducible and comparable across models.

How to use this guide

• Quick decision? Start with the Overall results leaderboard and Choosing a model.
• Latency-sensitive deployment? See Latency by provider — the serving provider changes latency as much as the model does.
• Matching a model to a kind of work? See Use-case types in the golden dataset — how each model handles document reconciliation vs provisioning vs batch loops, and happy paths vs edge, negative, and adversarial scenarios.
• Validating a model for a specific use case? Jump to the use-case breakdown.
• Understanding the numbers? See How to read the numbers and Methodology.

---

Models, providers, and effort

Each model was benchmarked at low reasoning effort. Accuracy figures come from the run on the model's primary serving provider (listed below); the same benchmarks were then re-run on an alternate provider to measure how serving infrastructure affects latency.

Higher reasoning-effort tiers will be added in a future benchmark round.

---

Overall results

Accuracy is the macro-averaged pass rate across all 6 AOPs and 7 tools. Latency is time-to-first-token (TTFT), averaged across the 6 AOP suites on the model's primary provider.

Choosing a model

Directional takeaways from this round:

• Highest accuracy on multi-step AOPs: GPT-5.5 led the AOP average (94.9%) and was the only model to fully pass New Hire Onboarding and 3-Way Matching in the same round.
• Fastest first token: the Claude family — Sonnet 4.6 on Anthropic (45.6s avg TTFT) and Opus 4.7 on Bedrock (50.5s) — started responding faster than GPT-5.5 on AOP workloads.
• Tools are commoditised, AOPs are not. All three models scored 83%+ on individual tools; the spread only opens up on multi-step AOPs (81–95%). Choose your model based on the AOPs you plan to run.
• Provider choice is a real latency lever — see the next section before you finalise infrastructure.

---

Latency by provider

We ran the same benchmarks on the same model across different serving providers. For GPT-5.5 we compared OpenAI with Azure OpenAI; for the Claude family, Anthropic with AWS Bedrock. Figures are average TTFT across the 6 AOP suites; each is a single run per provider, so read the gaps as directional.

Two patterns worth noting:

• GPT-5.5 was consistently faster on Azure OpenAI than on OpenAI in this round — 37.6s vs 52.6s average TTFT, with OpenAI slower on every AOP.
• The Claude family split: Sonnet 4.6 was faster on Bedrock, while Opus 4.7 was faster on Anthropic (48.4s vs 50.5s).

TTFT per AOP and provider

---

Use-case types in the golden dataset

The golden dataset is not a flat list of prompts — every scenario encodes a kind of work (the AOP's workload type) and a kind of test (the scenario type). Reading results along these two axes shows where models genuinely differ.

AOP workload types and complexity

The six AOPs fall into four workload types, ordered below from least to most complex. The mix column reads happy path · edge case · negative · adversarial.

Complexity reflects the structure of the workflow the agent must orchestrate — not how models scored on it:

• Moderate — a few tools in a linear, read-only flow (a fetch followed by analysis).
• High — several tools plus one demanding construct: multi-document reconciliation with tolerance rules, or a dozen query intents with role branching.
• Very high — many tools plus multiple constructs at once: approval gates, loops, generated code, and writes across several enterprise systems.

Structural complexity and model difficulty don't map one-to-one: the very-high-complexity onboarding and offboarding flows scored 78–100%, while the high-complexity LMS handler produced the round's lowest scores. Complexity tells you how much a run can go wrong; the pass rates tell you how often it did.

The dataset also mixes in multilingual triggers (Spanish and Portuguese variants of full flows), scripted multi-turn follow-ups (the test user answers the agent's questions mid-run), and file attachments (POs, invoices, GRNs) — so pass rates reflect complete conversations, not single prompts.

Accuracy by scenario type

AOP scenarios only (57), micro-averaged across all six AOPs on each model's primary provider.

Two Sonnet 4.6 runs where the grading service returned no result are excluded from its denominators.

Three things stand out:

• Happy paths are the hardest — for every model. That sounds backwards until you see what "happy path" means here: the longest end-to-end runs with the heaviest rubrics (a full onboarding with provisioning across four systems, or all twelve LMS query intents). Models rarely stumble on guardrails; they stumble on completing long flows perfectly.
• GPT-5.5 was flawless outside happy paths. Every one of its four misses was a full end-to-end run; it passed all 30 edge, negative, and adversarial scenarios.
• Guardrail behaviour is strong across the board. No model skipped an approval gate on request or acted on a planted false premise in provisioning flows. The few negative-type misses were failures to stop after a rejected ALL-approve gate (both Claude models, in Employee Offboarding).

Tool categories

The seven tools group into four categories. Pass rates are micro-averaged over the category's scenarios.

At the tool level the models are near-interchangeable: identical on retrieval and document work, with the only separation being three Opus 4.7 misses on data analysis and Sonnet 4.6 dropping six conversational workflow submissions (Self Service Tasks).

---

Results by use case — AOPs

Pass rate is the share of scenarios whose run satisfied the scenario's rubrics. TTFT and P95 total response are measured on the model's primary provider. All runs at low reasoning effort.

Invoice Query Assistant

Workload type: data query & analysis · Complexity: moderate. Fetches user invoices and uses data analysis to answer queries about invoice data. 8 scenarios — 5 happy path, 2 edge case, 1 adversarial. Tools used: Invoice Fetch API, Data Analysis.

Notable pattern: the "total amount due this year" scenarios — which require distinguishing issue date from due date — caused most failures (Sonnet 4.6 failed both variants; Opus 4.7 failed one). All models resisted the adversarial scenario where the user's claimed count contradicts the data.

3-Way Matching Purchase Order Reconciliation

Workload type: document-centric reconciliation · Complexity: high. Reconciles a Purchase Order, Vendor Invoice, and Goods Receipt Note via 3-way matching, flags discrepancies in an exception table, and routes major discrepancies for approval. 11 scenarios — 2 happy path, 6 edge case, 2 negative, 1 adversarial. Tools used: Document Parsing, Request Approval, Send Notification.

Notable pattern: the strongest AOP in the benchmark — all three models passed every scenario, including tolerance edge cases, currency mismatches, and the "missing GRN — must not fabricate a match" negative.

New Hire Onboarding and System Provisioning

Workload type: approval-gated provisioning · Complexity: very high. Automates end-to-end system provisioning for new hires (Azure AD, Jira, Salesforce, MS Teams, groups) with manager approval gates before provisioning begins. 7 scenarios — 2 happy path, 3 edge case, 1 negative, 1 adversarial. Tools used: Employee Directory Search, Request Approval, Create User in Azure AD / Jira / Salesforce, Add User to Group, MS Teams access, Send Notification.

Notable pattern: the only failures were on the full English happy path with the any-approve gate — GPT-5.5 was the only model to pass it. All models correctly stopped when the approver rejected, asked instead of fabricating a missing manager, and refused to skip approval gates on the adversarial prompt.

Employee Offboarding

Workload type: approval-gated provisioning · Complexity: very high. Handles IT offboarding end-to-end: confirms offboarding, raises asset deprovisioning tickets, routes the access-revocation plan through an ALL-approve gate, revokes Jira/Azure AD access, removes AD/DL group memberships, and produces a termination summary. 9 scenarios — 2 happy path, 5 edge case, 1 negative, 1 adversarial. Tools used include Asset Deprovision, Request Approval, Get User Jira / Delete User Jira, Azure AD Deactivate, Remove User From Groups, For Each Loop, Document Parsing.

Notable pattern: the hardest scenario was the negative one — "approver rejects the ALL gate; workflow must stop with no revocation" — failed by both Claude models. All three models refused the adversarial request to skip the approval gate.

LMS Learning Query Handler

Workload type: data query & analysis · Complexity: high. Handles employee and manager queries about learning programs, courses, completions, and overdue training — 12 use cases across self-service and manager perspectives. 13 scenarios — 12 happy path, 1 adversarial. Tools used: LMS, Data Analysis, JavaScript Code Executor, For Each Loop, Send Notification.

Notable pattern: the toughest AOP in the round. Three manager/self scenarios — "Manager — remind associates", "Manager — team started learning", and "Self — learning started" — failed on all three models. Straightforward self-service queries were near-universal passes.

MVR Check

Workload type: batch loop processing · Complexity: very high. Runs Motor Vehicle Record checks for eligible recruitment candidates: fetches candidates, computes eligibility with generated code, branches contact logic between SMS and email, and updates attempt counters in a loop. 9 scenarios — 4 happy path, 4 edge case, 1 negative. Tools used: Fetch User Details, Fetch Form Link, JavaScript Code Executor, Reminder, Update Attempts, For Each Loop.

Notable pattern: loop resilience separates the field — "candidate record missing a Job Application ID mid-loop" failed on both Claude models; GPT-5.5 was the only model to handle it. Opus 4.7 also acted on a read-only status query that should not have triggered sends. All three refused the consent-override request.

---

Results by use case — Tools

Beyond multi-step AOPs, each individual tool is exercised in isolation and graded the same way. Latency for tools is average TTFT on the model's primary provider; effort is low throughout.

Tool latency (Avg TTFT)

What each tool does

• Case Management — creates, views, and manages support cases/tickets, including on behalf of another employee, filling case forms conversationally.
• Data Analysis — filters, aggregates, groups, and summarises structured data (CSV/Excel/JSON) fetched earlier in the run.
• Document Parsing — extracts text, key fields, and tables from uploaded documents (PDFs, images).
• Employee Directory — looks up employee records to answer people queries and resolve identities (manager, department, designation, contact).
• JavaScript Code Executor — generates and executes JavaScript for calculations, data transformations, string/regex processing, and date math.
• Orders & Products List View — fetches tabular data from a workflow-backed list view with exact-match filters and free-text search.
• Self Service Tasks — runs a workflow conversationally (Leave Request, Payslip Retrieval, Employee Termination, Employee Information Update, Daily Check-In), collecting required fields turn-by-turn and submitting without a separate confirmation step.

---

How to read the numbers

Each scenario is graded by an independent LLM judge that returns pass/fail on a set of criteria. The metrics reported are:

Grading criteria

---

Methodology

• Golden dataset. 118 curated scenarios — 57 across 6 AOPs and 61 across 7 tools — each defining the user input, expected tool calls, expected behaviour, and explicit pass criteria. The mix deliberately includes happy paths, edge cases, negatives (the agent must refuse or stop), and adversarial prompts (the user tries to bypass controls or plant false premises).
• Deterministic tool mocks. All tool outputs are mocked and deterministic, so every model sees identical data and results are reproducible and comparable.
• Independent LLM judge. Every run is graded by an external LLM judge against the scenario's rubrics; scenarios where the grading service returned no result are excluded from the pass-rate denominator (a harness failure, not a model failure).
• Reasoning effort. All models ran at low reasoning effort. Higher effort tiers are planned for a future round.
• Latency measurement. TTFT and total response time are measured across the agent's full turn in a production-equivalent environment, not isolated API calls — they include orchestration, tool execution, and multi-call agent loops, which is why figures are higher than raw model TTFT.
• Single-run caveat. Each model × provider × use-case cell is one run over a small scenario set. Treat small deltas as noise; the results are directional.
• Model coverage. This page reports the models currently selectable on the platform. The benchmark round also covered additional frontier models under evaluation; full results for every model are in the open benchmark repository.
• Benchmark date. July 2026. Model providers update capabilities continuously; this page is re-run and updated periodically.

Per-scenario results, full scenario definitions, pass criteria, and the exact AOP instructions are published in the open benchmark repository: aic-benchmarks.

---

Glossary