Token Management & Efficiency in the Orchestration Layer

When Leena acts as the orchestration layer for AI Colleagues that customers build themselves — particularly when those AI Colleagues connect external MCP servers — a reasonable concern follows:

If we can connect our own tools and MCP servers, what stops token consumption from running away? How do we know the orchestration layer isn't inflating every prompt?

Token efficiency in Leena is not a best-effort guideline. It is engineered into six layers of the orchestration runtime, most of it enforced by hard limits rather than by instructions to the model.

1. Model routing by task complexity

Every LLM call in the orchestration runtime is classified by task complexity — light, medium, heavy, or reasoning-heavy — and routed to the cheapest model tier that can perform it.

Routing decisions, orchestration classification, summarisation, and utility calls run on lightweight models. Only genuinely complex reasoning steps use heavyweight or reasoning models. Per-use-case model mappings, with ordered fallbacks, are centrally configured, so the platform never spends premium-model tokens on routine steps.

2. Context engineering — keeping the window small

This is the core of how Leena AI keeps AI colleagues efficient when MCP servers are connected.

Conversation windowing and summarisation

Context is actively managed against a token ceiling using either sliding-window retention basis time — system context plus the most recent N messages — or LLM summarisation of older history using a lightweight model. Long-running conversations do not accumulate unbounded context.

Large tool and MCP outputs never flood the context

Tool and connector outputs beyond a size threshold are automatically offloaded to artifact storage. The model receives a compact, structure-preserving preview — the first items of arrays, the first keys of objects, truncated strings, with explicit truncation markers — plus a reference handle. It retrieves only the specific slices it actually needs, on demand.

Code-execution flows use a reference protocol so large datasets are manipulated by reference rather than being serialised through the prompt.

Curated tool surface per turn

Tool and AOP definitions consume prompt tokens, and a single MCP server can expose dozens of tools. The platform only exposes the tools relevant to the current turn — filtered by permissions and scope, then selected for relevance by a reranking step — with an explicit budget on how many tool definitions may be exposed to the model at once.

This is the same filtering that underpins the security model. See Prompt Injection Defence.

3. Prompt caching

System prompts and tool definitions are assembled with a deliberate stable-prefix strategy: static content first, deterministic ordering, session-specific content appended last. The assembled prefix is stamped with provider cache-control markers, so provider-side prompt caching applies to the unchanged prefix on every subsequent turn.

Cache reads are billed by providers at a fraction of normal input cost. Cache read and write tokens are tracked per call, so the savings are measurable rather than assumed.

4. Hard caps and loop protection

Efficiency is enforced, not just encouraged.

Per-call output caps. A hard limit on output tokens for any single LLM request — 10,000 tokens in the orchestration runtime — with tier-based expected-output budgeting below that.

Per-turn execution caps. Bounded steps and tool calls per turn, bounded parallelism, and bounded reasoning-token budgets for thinking models.

Loop protection. Iteration ceilings, per-run and total time ceilings, and repeated-action detection: an execution that issues the same action repeatedly is terminated. A misbehaving or manipulated execution cannot burn tokens in a loop.

5. Rate limiting and provider efficiency

Distributed token buckets per model deployment. Before each call, the platform estimates the request's token cost — actual prompt tokens via tokeniser counting, plus expected output — and atomically consumes them from a Redis-backed bucket shared across all pods. After the call, the bucket is reconciled against actual usage. A depleted deployment is skipped in favour of the next candidate instead of queuing failures.

Circuit breakers and fallback chains. Failing model deployments are detected within a sliding window and taken out of rotation, so tokens and latency are not wasted on doomed retries. Provider → model → deployment fallback keeps requests on healthy capacity.

Waste avoidance. Abandoned requests are aborted rather than completed. Terminal failures, such as content-filter stops, are not retried blindly. Truncated free-text outputs are recovered as partial results rather than regenerated from scratch.

6. Metering and visibility

Every LLM call is persisted with its model, deployment, token usage (prompt, completion, total), latency, and status. Usage is aggregated per request and per conversation turn, converted to cost, and exported to platform metrics including cache hit and miss rates.

Token consumption is available to administrators in dashboard reporting, broken down by AOP and by AI Colleague — so you can see exactly where tokens are spent and validate the mechanisms above against your own real usage rather than taking them on trust.

See Analytics → AI Colleagues for the reporting surface.