How Leena AI decides run status
Every run in Run history and on the AI Colleagues dashboard has one status: Handled, Unhandled, In progress or Cancelled. This page explains when a conversation becomes one run or several, how each status is decided, and what happens in common follow-up scenarios.
When a message starts a new run
A run is one request and everything done to complete it: replies, AOPs, Tool calls (inc knowledge), forms and sign-ins. Each run has its own Run ID.
When a user sends a message, the platform checks the latest run in that conversation:
| Latest run in the conversation | What the new message does |
|---|---|
| Waiting for the user: the assistant asked a question, needs a missing detail or a confirmation, or a form or sign-in is open | Continues the same run (same Run ID) |
| Stopped by the user with the Stop button, within the last hour | Continues the same run |
| Finished (Handled, Unhandled or Cancelled), or there is no earlier run | Starts a new run (new Run ID) |
The topic of the message does not change this. The platform does not check whether a follow-up is related to the previous question; only the state of the latest run matters. Within the same conversation, a new run still sees the earlier messages, so a related follow-up is answered in context.
flowchart TD
M[User sends a message] --> Q{Is the latest run<br/>waiting for the user?}
Q -- Yes --> C[Continue that run]
Q -- No --> N[Start a new run]
C --> R[Assistant replies]
N --> R
R --> W{Does the reply need<br/>the user's input?}
W -- Yes --> P[Run stays In progress]
W -- No --> G[Run finishes and is graded]
When a run finishes
After each reply, the platform decides whether the run is finished:
- It stays In progress when the reply needs something from the user: a clarifying question, a missing detail, a confirmation before an action, a form to fill in, or a sign-in.
- It finishes when the reply answers the question, completes the task, gives the user a form or link to complete it, or says the request cannot be handled.
When a reply ends with an offer, an LLM check makes the call. An answer followed by an offer such as "Is there anything else I can help with?" or "Would you like me to raise a ticket?" normally finishes the run. A confirmation the assistant needs before it acts, such as "Shall I submit this request?", keeps it open.
If the user does not reply. A run that is waiting for the user closes automatically 1 hour after the user's last message. It is Handled if the user had already been given what they asked for; otherwise it is Cancelled.
Scheduled (Workbench) and API runs are not split by messages: each trigger is one run, graded the same way when it finishes.
How Handled and Unhandled are decided
When a run finishes, its status is set in this order:
- Errors and stops. A run that ended in a system error or timeout is Unhandled. A run stopped by the user, cleared from chat, cancelled by an admin, or left by the user mid-way is Cancelled. Most guardrail blocks also end the run as Cancelled.
- Single knowledge-base answers. If the run was one question answered by one knowledge-base search, it is Handled when the answer cites sources and Unhandled (Knowledge gap) when no answer was found.
- Everything else. An LLM review reads the messages in the run, together with the result of every Tool and AOP call in it, and decides whether the user's original request in that run was met.
The review only sees the run being graded, not earlier runs in the conversation. It treats these as Handled:
- A clear answer, including a "no" that answers the question, such as "You are not eligible for this leave"
- A completed task, such as a leave applied, or a ticket raised when the user asked for one
- A form or link the user can use to complete the task
- A policy explained, even if the outcome is not what the user wanted
- A greeting answered
- An escalation resolved with an outcome, approved or rejected
And these as Unhandled:
- "I don't know" or "I couldn't find that"
- A generic fallback instead of an answer, such as "Please raise a ticket" or "Contact HR"
- A request outside what the AI Colleague is set up to do
- The assistant still asking for details it needs
- An unclear or incomplete answer
- A Tool or AOP failure that nothing recovered from
Tool results count as fact. If a reply sounds successful but the Tool call behind it failed, the run is Unhandled, and a run in which every Tool call failed is always Unhandled. A failure that a later step recovered from does not count against the run.
Why a run is Unhandled
Each Unhandled run gets one category, shown in Unhandled due to, with the full explanation under View reason:
| Category | Meaning |
|---|---|
| Guardrail | A safety or policy check blocked the request or the reply |
| Tool / AOP failure | A Tool or AOP step failed, for example an API or validation error |
| Knowledge gap | The knowledge base had no answer to a work question |
| Out of scope | The request is outside what the AI Colleague is set up to do |
| System error | A technical error or timeout, or any case the categories above do not cover |
When more than one category applies, the one higher in this table is used.
Follow-up scenarios
Each scenario shows the usual path first. Where the LLM review makes the call, the status shown is the typical result.
1. Answered, then a follow-up the assistant cannot answer, then "raise a ticket"
"What is the leave carry-forward limit?" → answered
"Can I carry forward sick leave?" → "I couldn't find that."
"Raise a ticket."
Usually three runs:
| Run | Status |
|---|---|
| Carry-forward question, answered | Handled |
| Sick-leave question, no answer found | Unhandled (Knowledge gap) |
| "Raise a ticket", ticket created | Handled |
If the assistant was still waiting for the user when they asked for the ticket, for example because an AOP step asked "Shall I raise a ticket for you?", the ticket request joins the second run. That run is graded on its original question, which was not answered, so it is normally Unhandled (Knowledge gap) even though the ticket was raised.
If the ticket cannot be created, the run that tried to create it is Unhandled (Tool / AOP failure).
2. Answered, then "raise a ticket"
"How do I update my bank details?" → answered
"Raise a ticket for this."
Usually two runs: the question (Handled) and the ticket request (Handled when the ticket is created, Unhandled with Tool / AOP failure when creation fails). If the assistant shares a pre-filled ticket form instead of creating the ticket, the ticket run is still Handled, because the user has what they need to finish it.
If the first run was still waiting for the user, the ticket request joins it and the run is graded as a whole. The question was answered and the ticket was raised, so it is Handled. If the ticket creation failed, that failed Tool call normally makes the run Unhandled.
3. A follow-up on the same topic
"What is the leave policy?" → answered
"How many casual leaves do I get?"
Two runs when the answer finished the first run, which is the usual case. Each run is graded on its own question. The second run sees the first exchange, so the follow-up is answered in context.
One run when the follow-up arrives while the first run is waiting for the user, for example as part of the reply to "Which location are you in?".
4. A follow-up on a different topic
"What is the leave policy?" → answered
"Reset my VPN password."
Two runs when the first run had finished. This is the same rule as scenario 3: a change of topic is not treated differently.
One run when the first run was still waiting for the user, for example in the middle of an AOP that is collecting details. The new request is passed to the assistant and handled inside the same run, and the AOP's progress is kept so the user can return to it. The run is graded as a whole, so if the original task is left unfinished, the run can be Unhandled even when the new request was handled.
Other cases
| Situation | What happens |
|---|---|
| The user stops a reply with the Stop button | The run shows Cancelled. If they send another message within the hour, the same run continues. |
| The user leaves while the assistant waits for them | After 1 hour: Handled if they already had what they asked for, otherwise Cancelled |
| The user clears the chat | The open run is Cancelled |
| An admin cancels a run from Run history | Cancelled, with the admin's name and note |
Escalations and tickets
Escalation is a separate flag, not a status. It is set to Yes only when the run is escalated to an escalation contact. Raising a ticket in your ticketing or case system is a normal Tool or AOP action: it does not set Escalation, and it makes a run Handled only when a ticket is what the user asked for in that run.
Reading the numbers
- One conversation often contains several runs, and each one counts in Total runs and the status gauges.
- "Couldn't answer, so raise a ticket" usually shows up as an Unhandled (Knowledge gap) run followed by a Handled ticket run. Filter Unhandled runs on Knowledge gap to find questions to add to the knowledge base.
- To see why a run was marked Unhandled, open it in Run history and select View reason.
- The Handled requests gauge in Bot Analytics counts individual messages, not runs, so its numbers differ from this dashboard. See the Glossary.
Related pages
- Run history
- AI Colleagues analytics
- Run Execution Details
- How Leena AI Chooses the Response Language, which uses the same rule for when a request stays open
Updated about 12 hours ago
