Evals

Cases run one after another, sharing one runtime. A case that times out, crashes, or hits an agent error fails with that error as its reason. Each tool call is recorded with its arguments, whether it succeeded, its duration and a preview of up to 500 characters of its result, on every runtime.

Body. The ## Prompt section is sent to the agent. ## Criteria turns on the judge. Headings match at any level and without regard to case, and Input or Ask work for the prompt, Rubric, Expect or Expected for the criteria. Text before the first of these headings is a note for people and is ignored. A body with no headings at all is the prompt, so a one-line file is a valid case. A body with criteria but no prompt is an error.

Tool names. Write the short name, read_knowledge, not mcp__hubzoid__read_knowledge. Names are compared without the mcp__hubzoid__ prefix and without regard to case.

What is checked. Text checks run against the agent's final answer only. Tool activity lines and the thinking panel are removed first, so not_contains never trips over the agent's own reasoning. Tool checks look at every call the case made.

Folders. Cases can live in subfolders. A case's name is its path under evals/ without .md, so evals/approvals/reply-reject.md is approvals/reply-reject, and --case 'approvals/*' selects the folder. Files and folders starting with _ or ., and the evals/fixtures/ folder, are not cases, so drafts, notes and test data can live beside them.

---
timeout: 180
expect_tool_args:
  read_knowledge:
    name: house-rules
---
## Turn 1
What is the refund policy?

## Turn 2
Does that apply to an order delivered 40 days ago?

## Criteria
Uses the actual policy and keeps the context of the first answer.

contains, not_contains and the judge look at the final reply. Tool checks cover the calls of every turn. A case cannot have both ## Prompt and ## Turn sections, and a turn with no text is an error.

For example, on a Hub with lookup_order and send_email tools:

---
fake_tools:
  - tool: lookup_order
    args: {order_id: A-1001}
    output: {status: delivered, delivered_on: "2026-09-14"}
  - tool: send_email
    error: SMTP is down
real_tools: [read_knowledge, days_since_delivery]
---
## Prompt
Can order A-1001 still be refunded?

• The first fake whose tool and arguments match answers the call. Arguments match as in expect_tool_args.
• Any other call of a Hub tool that is not in real_tools is blocked. The agent reads "Blocked in this eval", nothing runs, and the case fails naming the call. A missing fake can never cause a real effect.
• An error fake is a failed call, as if the tool had raised that message.
• Access checks and approval rules still apply. A restricted tool's access check runs before its fake.
• MCP servers do not pass through Hubzoid's tool layer, so they are not covered. A faked case on a Hub with MCP servers fails unless it sets allow_real_mcp: true. Do that only when those servers just read.

It sees:

• the Hub's AGENTS.md instructions, so the rules you wrote there apply to every judged case without restating them,
• the case prompt, or every turn of a conversation, and the criteria,
• the answer,
• the tools the agent actually called, with their arguments, and the tools the Hub has, as observed facts.

The runtime section Hubzoid appends to your instructions is not shown to the judge, so it grades your Hub rather than the framework. A judge that errors or returns no score fails the case with the reason judge failed: ..., which is easy to tell apart from a low score.

Results. Every run writes a JSON file to .hubzoid/evals/, named after its start time with a unique suffix, and updates the index.json summary there. The file records how the run started, the model under test and, for each case, its verdict, reason, duration, checks, judge score and reasoning, turns, tool calls with their arguments, and full response. The newest 200 runs are kept. HUBZOID_EVAL_KEEP_RUNS changes that, and 0 keeps every run. Saved arguments, previews, errors, transcripts and the judge's reasoning are redacted for values that look like secret keys and tokens. That cannot catch every secret written in prose, so use synthetic data and protect the Hub's backups. Keep .hubzoid/ out of Git.

When CI=true is set, as most CI systems do, the run is recorded as started by CI. When a markdown task commits and pushes changes to knowledge/, that push triggers your CI like any other, so unattended changes are evaluated too.

For example, add two lines to the frontmatter of the starter's case and keep its body:

---
schedule: "0 6 * * 1"
tags: [canary]
contains: ["house-rules"]
---
## Prompt
A customer wants a refund 40 days after delivery. What can we offer?

## Criteria
Follows the refund rule in knowledge/house-rules.md for that many days,
and names house-rules.md as its source.

Scheduled evals use the same scheduler as markdown tasks: the same 5-field cron in the machine's local time, the same catch-up once after downtime, and the same idle gate, so an eval never starts while someone is mid-conversation. All cases due at the same time run as one suite on the Hub's workflow engine, so five cases on one cron start the runtime once.

A scheduled suite with a failing case saves its results first, then ends as a failed run. It is logged at ERROR, shows in hubzoid eval status and Settings, and raises a workflow alert when alerts are configured. See operating runs. HUBZOID_DISABLE_SCHEDULE=1 stops scheduled tasks and scheduled evals together.

It opens with the score: the latest run's passed cases out of the total, its time and how it started, a bar for each of the last 10 runs, the cases failing now and the cases never run. Below that are the cases, each with its last 10 results, the run history and each run's details. On the Agents list, an agent with cases shows its latest score, such as Evals 8/10, or Evals not run.

Case names and pass or fail results are shown to everyone who can see the tab, but the details of a run made with run_as belong to that account only. Other viewers see a notice that the result is private.

It shows ok when its latest result passed for the case and the Hub as they are now, fail when it failed, and pending when it never ran, ran with --no-judge although it has criteria, or the case, the instructions, skills, knowledge, tools, workflows, settings or model changed since. A requirement that matches no case fails, so a typo never reads as satisfied.

HUBZOID_OTEL_ENDPOINT=https://langfuse.example.com/api/public/otel
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...

The keys are also read from OTEL_EXPORTER_OTLP_HEADERS when tracing is set up with Basic auth there. Each Hub gets a dataset named hubzoid-evals-<hub>. Each case becomes a trace with a pass score, one score per check, and the judge's score with its reasoning. Every eval trace carries the tag hubzoid.eval, so you can keep eval traffic out of production dashboards. The push is best effort: a Langfuse outage never fails a run, and the local JSON remains the record.

One agent run makes one model request, plus one more each time it calls tools and reads the results. A multi-turn case runs the agent once per turn.

• Use free checks for facts, criteria for judgement. contains suits a figure or phrase that must appear verbatim. Anything that depends on wording or context belongs in ## Criteria. When a free check and the judge disagree, the free check is usually the one that is wrong.
• Check arguments, not only calls. expect_tool_args proves the agent asked for the right record, not just that it called the tool.
• Make criteria cover every branch. If a tool can return data, a link or an error, say what a correct answer looks like in each case.
• Say what a failure is. "Guessing the number of days is a failure" grades more consistently than "reports the number of days".
• Put shared rules in AGENTS.md. The judge already grades against it, and the agent follows it in production, not only in evals.
• Mind real systems. Evals run with the Hub's real credentials and tools. A case that calls a write-capable tool really writes. Prefer fake tool results, read-only prompts and test accounts, or point scheduled evals at a Hub configured against a sandbox.

The account must be usable: it has signed in, is not blocked or awaiting approval, and holds Use this agent. A case's own run_as wins over --run-as. Because it grants nothing, a case run as an account that holds the permission proves the tool works for the people who should have it, and a case run as one that does not proves it stays refused.

On every runtime the agent is not shown a tool the person may not use, and a call that names it anyway is refused. Either way the answer is what matters, so check it with criteria rather than forbid_tools. See restrict tools.