How We Benchmark AI Colleagues

Why this page exists

Most enterprise software categories converge on a shared way to measure quality — throughput benchmarks for databases, uptime SLAs for cloud providers, common threat frameworks for security tooling. Agentic AI does not have an equivalent yet. Vendors, including us, make accuracy and reliability claims that are rarely independently reproducible, because there is no shared definition of what "accurate" means for a system that reads a document, decides which tool to call, executes a multi-step workflow, and then explains what it did.

This page documents how AI Colleagues are evaluated internally: what we test, how scenarios are graded, what we measure, and how the results feed back into product configuration decisions. The benchmark itself is open source, so anything described here can be inspected and re-run.

Repository: github.com/shubham-leena-ai/aic-benchmarks

---

Why agentic evaluation is different

Evaluating a traditional software feature is input → output → compare against an expected value. An agent doesn't work that way: its job is to make a sequence of judgment calls, not to return one deterministic string.

Take purchase order reconciliation. The agent has to parse a PO, an invoice, and a receipt; identify discrepancies; decide whether a discrepancy falls inside tolerance; and then either approve the match or escalate it with the right context attached. There is no single correct output to diff against. What matters is whether the agent did the right things, in a defensible order, reached the right conclusion, and left an auditable trail explaining why.

The same holds across most of the workflows AI Colleagues handle in production — onboarding a new hire and provisioning their systems, answering a benefits question correctly and in the employee's own language, running a background check and flagging the right exceptions, resolving a learning-management query without inventing policy that doesn't exist.

A demo can't surface any of this. A demo is a curated path through best-case behaviour. It says nothing about the requests with a missing field, ambiguous phrasing, or a workflow that has to pause and resume across turns.

Evaluating this honestly requires two things: a large, varied set of realistic scenarios, and a way to grade judgment rather than output strings.

---

What we evaluate: two layers

Evaluation is split into two layers because they fail in different ways and have to be measured differently.

1. Agentic Operating Procedures (AOPs)

Multi-step, tool-using business workflows the agent has to complete end to end. Each one chains multiple tool calls, decision points, and — frequently — a handoff between automated steps and human approval.

Current AOP coverage:

• Purchase order reconciliation
• New hire onboarding and system provisioning
• Employee offboarding
• Invoice query handling
• Background check processing
• Learning management queries

2. Tools

The individual capabilities an agent draws on inside those workflows — document parsing, employee record lookup, calculation, code execution, case management.

A tool can behave perfectly in isolation and still fail inside a workflow if the agent invokes it at the wrong point or misreads its output. That's precisely why testing tools alone is insufficient, and why both layers are tested separately and together.

---

Reproducibility by design

Every scenario runs against deterministic, mocked tool outputs.

This matters more than it first appears. If a test environment calls live systems, results get noisy for reasons unrelated to the model: a flaky API, a slow network hop, a record that changed between runs. Mocking tool outputs means every model sees identical inputs — so when one model passes a scenario and another fails it, the difference is real and attributable to the model, not to what each run happened to encounter.

This is what makes results comparable across models, and what allows anyone to clone the repository, re-run a scenario, and trace a failure back to the exact input that caused it.

---

How runs are graded

Each scenario is graded by an independent LLM judge against explicit criteria, split into two categories.

Execution-behaviour criteria gate the pass/fail decision. Did the agent call the right tools, in a defensible order, and reach the right outcome? For the purchase order case: did it parse all three documents, correctly identify the discrepancy, and request approval before proceeding? These are authored per workflow, because "correct" looks different for onboarding than it does for a benefits question.

General-quality criteria are graded alongside every scenario, regardless of workflow.

A scenario passes only when its execution-behaviour criteria are satisfied. Quality criteria are reported alongside rather than gating, because a technically correct answer delivered in the wrong language — or padded with hallucinated detail — is not a passing experience for the end user.

Grading criteria reference

---

Metrics we report

---

Current coverage and how to read results

At the time of writing, the benchmark has been run across five frontier models — GPT-5.5, GPT-5.6 Sol, GPT-5.6 Terra, Claude Sonnet 4.6, and Claude Opus 4.7 — each at low reasoning effort, across 6 AOPs and 7 tools, all against deterministic mocked tool outputs.

Read gaps with care. Each cell in the published tables is a single run over a small scenario set (7–13 per AOP, 5–11 per tool). Differences of a few points sit inside run-to-run noise. Treat these as directional signals, not a definitive ranking.

Live numbers are maintained in the repository, not in this page. Model rankings shift every few months; any table pasted into docs will be stale within a quarter. See github.com/shubham-leena-ai/aic-benchmarks for current results.

What we expect to remain true, and state confidently: no single model wins across every workflow and every tool. Different models show different strengths by task type, and the gap between top performers is often smaller than the gap created by provider routing choice for the same model. That pattern has held steady even as specific rankings shifted underneath it.

---

The provider layer matters as much as the model

The most durable finding from this work is architectural rather than model-specific.

We expected model choice to be the dominant performance variable. It wasn't the whole story. The serving provider — OpenAI vs. Azure OpenAI, Anthropic vs. AWS Bedrock — mattered just as much, and sometimes more.

• Every GPT model tested was substantially slower on Azure than on OpenAI directly — 35% to 54%, depending on the model.
• The Anthropic picture was mixed: Sonnet 4.6 was slower on Bedrock than on Anthropic directly, while Opus 4.7 was faster on Bedrock.

That inconsistency is itself the useful data point. Provider effects are real and often large, but they are not a universal law you can assume applies to every model. They have to be measured per model, not taken on faith.

Why this matters for deployment architecture: provider choice is usually treated as a procurement or compliance decision — driven by existing cloud commitments or data residency requirements. This data says it should be treated as a performance decision too. The same model, same weights, same prompt, can feel meaningfully slower to your employees purely because of the network path the request takes.

This finding is about infrastructure, not about which model currently tops a leaderboard, so we expect it to hold across model generations.

---

How the benchmark feeds product decisions

This isn't an academic exercise — it was built because we needed it.

Earlier this year, customer feedback indicated AI Colleague was responding more slowly than it should. The benchmark gave us a rigorous way to test whether a different model or provider configuration would resolve it, without guessing. The data pointed to a configuration change that improved both accuracy and latency, and we shipped it.

That specific decision is already superseded — a newer model has since become available and been re-tested. That is the point. The benchmark is the durable asset; any single decision it produces has a shelf life.

---

Appendix: questions worth asking of any agentic AI benchmark

Applies to ours as much as anyone's. If you are evaluating or operating an enterprise agentic AI platform, "is your AI accurate?" isn't a useful question — every vendor says yes. More useful:

• Can you see the actual scenarios tested against, not just a summary claim?
• How is a multi-step workflow graded, as opposed to a single response?
• Do published results run against reproducible conditions, or a demo environment that can't be inspected?
• How is latency measured, at what percentile, and has serving infrastructure — not just the model — been tested as a variable?
• What is the process when the underlying model changes, and how is regression verified?

The benchmark is open at github.com/shubham-leena-ai/aic-benchmarks. It can be read, re-run to verify results independently, or forked and adapted to test agentic AI vendors — including us — against workflows that matter to your organisation.