--- title: Evals description: Check that a Hub keeps doing its job with Markdown eval cases, free checks, a model judge, scheduled runs, CI gates and optional Langfuse history. canonical_url: https://hubzoid.com/docs/guides/evals last_updated: 2026-09-27 --- # Evals Check that a Hub keeps doing its job with Markdown eval cases, free checks, a model judge, scheduled runs, CI gates and optional Langfuse history. An eval asks the Hub's agent a question and checks the answer. You write one Markdown file per behaviour you care about, and `hubzoid eval run` runs them all through the Hub's own runtime, with the same model, tools, MCP servers and access rules as real traffic. The exit code makes the same command a CI gate. ## When to use evals - Before you change `AGENTS.md`, a skill or a tool, to see what the change breaks. - On every push in CI, as a gate. - On a schedule, to catch drift you did not cause, such as a provider updating a model or a data source changing shape. - To prove access rules hold, for example that a restricted tool is refused to people without the permission. ## The smallest eval ```markdown title="evals/refund-window.md" ## Prompt What is the refund window for a cancelled order? ## Criteria States 14 days. Does not invent an exception process. ``` That is a complete case. The file name without `.md` is the case name. A case with a `## Criteria` section is graded by a model. A case without one runs only the free checks you declare. There is no separate switch for the judge. ```bash hubzoid eval run ./ops-desk ``` ```text 3 case(s) · ops-desk judge: claude-local (hub default — pin it for stable scores) ✓ refund-window ✗ stock-lookup never called: stock_level ✓ escalation-policy case judge time reason refund-window PASS 9/10 4.2s stock-lookup FAIL — 3.8s never called: stock_level escalation-policy PASS 8/10 5.1s 1 failed, 2 passed of 3 ``` ## Case file format Every frontmatter key is optional. ```markdown title="evals/stock-lookup.md" --- schedule: "0 6 * * 1" tags: [canary] expect_tools: [stock_level] forbid_tools: [http_get] contains: ["north"] not_contains: ["as an AI"] timeout: 90 threshold: 7 enabled: true --- ## Prompt How many LMP-2041 lamps are in the north store? ## Criteria Reports the number the stock tool returned for the north store. Inventing a number, or answering for a different store, is a failure. ``` | Key | Type | Default | Checks | | --- | --- | --- | --- | | `expect_tools` | list | none | Each tool must be called at least once. | | `forbid_tools` | list | none | None of these tools may be called. | | `contains` | list or string | none | Each text must appear in the answer, ignoring case. | | `not_contains` | list or string | none | None of these texts may appear, ignoring case. | | `timeout` | seconds, 1 to 3600 | `120` | A case that runs longer fails. It never hangs the suite. | | `threshold` | 1 to 10 | `7` | The judge's pass mark. | | `tags` | list or string | none | Labels for `--tag`. | | `schedule` | 5-field cron | none | Run the case on a schedule inside the running Hub. | | `enabled` | boolean | `true` | `false` skips the case. | An unknown key is an error, not a warning, so a typo such as `expected_tools:` cannot make a case pass while checking nothing. **Body.** The `## Prompt` section is sent to the agent. `## Criteria` turns on the judge. Headings match at any level and without regard to case, and `Input` or `Ask` work for the prompt, `Rubric`, `Expect` or `Expected` for the criteria. Text before the first of these headings is a note for people and is ignored. A body with no headings at all is the prompt, so a one-line file is a valid case. A body with criteria but no prompt is an error. **Tool names.** Write the short name, `read_knowledge`, not `mcp__hubzoid__read_knowledge`. Names are compared without the `mcp__hubzoid__` prefix and without regard to case. **What is checked.** Checks run against the agent's answer only. Tool activity lines and the thinking panel are removed first, so `not_contains` never trips over the agent's own reasoning. Files starting with `_` or `.` are ignored, so drafts and notes can live beside the cases. The full list of Hub folders is in [Hub files](https://hubzoid.com/docs/reference/hub-files). ## How a case runs Run the agent → Free checks → Judge (if criteria) → Verdict The judge runs only when every free check has passed. 1. The agent answers the prompt in a chat of its own, `eval-`, so files never leak between cases. 2. The free checks (`expect_tools`, `forbid_tools`, `contains`, `not_contains`) run. They add no model call. 3. If they all pass and the case has criteria, the judge scores the answer. Cases run one after another, sharing one runtime. A case that times out, crashes, or hits an agent error fails with that error as its reason. ## The judge The judge is a single model call with no tools. It sees: - the Hub's `AGENTS.md` instructions, so the rules you wrote there apply to every judged case without restating them, - the case prompt and criteria, - the answer, - the tools the agent actually called and the tools the Hub has, as observed facts. It scores from 1 to 10, and the case passes at `threshold` or above. The runtime section Hubzoid appends to your instructions is not shown to the judge, so it grades your Hub rather than the framework. A judge that errors or returns no score fails the case with the reason `judge failed: ...`, which is easy to tell apart from a low score. **Pin the judge model** for anything you track over time. It defaults to the Hub's own model, which needs no setup. A model tends to rate its own output generously, and if the Hub's model changes, the scale changes with it. ```bash title=".env" HUBZOID_EVAL_JUDGE_MODEL=claude-local/opus ``` `--judge-model` sets it for one run and wins over the variable. ## Run and inspect ```bash hubzoid eval run ./ops-desk # every enabled case hubzoid eval run ./ops-desk --tag canary # cases with a tag hubzoid eval run ./ops-desk --case "stock-*" # cases matching a glob hubzoid eval run ./ops-desk --no-judge # skip the grading call hubzoid eval run ./ops-desk --model openai/gpt-4o-mini # try another model hubzoid eval run ./ops-desk --compare # also show what changed hubzoid eval run ./ops-desk --quiet # summary and failures only ``` | Command | Shows | | --- | --- | | `hubzoid eval list ` | Each case, what it checks, its tags and its schedule. | | `hubzoid eval status ` | The last run, its pass rate and what is failing now. | | `hubzoid eval explain ` | For one case from the last run: the prompt, the full response, the tools called, every check's verdict, the judge's score and reasoning, and the paths of the case file and `AGENTS.md`. | When a case fails, start with `explain`. Editing instructions in `AGENTS.md` or a skill, not code, is usually the fix. **Results.** Every run writes `.hubzoid/evals/.json` with the model under test and, for each case, its verdict, reason, duration, checks, judge score and reasoning, tool calls and full response. The last 50 runs are kept. This file is the durable record and the CI artifact. **Comparing runs.** `--compare` diffs the new run against the previous one and prints only what moved: ```text REGRESSIONS: 1 stock-lookup PASS → FAIL never called: stock_level refund-window FAIL → PASS ``` Cases match by name, so renaming a case file shows as one removed and one added, not as a regression. ## In CI The exit code is `1` when any case fails, `2` when a case file cannot be parsed, and `0` otherwise, including when there are no cases. ```yaml title=".github/workflows/evals.yml" - run: hubzoid eval run ./ops-desk --no-judge # every push - run: hubzoid eval run ./ops-desk # on main, judged ``` Keep `.hubzoid/evals/*.json` as a build artifact. When a markdown task commits and pushes changes to `knowledge/`, that push triggers your CI like any other, so unattended changes are evaluated too. ## On a schedule Give a case a `schedule:` and it runs inside the running Hub, without CI: ```markdown title="evals/refund-window.md" --- schedule: "0 6 * * 1" tags: [canary] --- ## Prompt What is the refund window for a cancelled order? ## Criteria States 14 days. ``` Scheduled evals use the same scheduler as [markdown tasks](https://hubzoid.com/docs/guides/markdown-tasks): the same 5-field cron in the machine's local time, the same catch-up once after downtime, and the same idle gate, so an eval never starts while someone is mid-conversation. All cases due at the same time run as one suite on the Hub's workflow engine, so five cases on one cron start the runtime once. A failing scheduled suite is logged at `ERROR`, so it reaches the logs you already watch, and shows in `hubzoid eval status`. `HUBZOID_DISABLE_SCHEDULE=1` stops scheduled tasks and scheduled evals together. A few canary cases on a weekly cron, and the rest in CI, keeps the cost predictable. ## Langfuse history Everything above works with nothing else installed. If the Hub already sends traces to Langfuse, eval runs are pushed there too: ```bash title=".env" HUBZOID_OTEL_ENDPOINT=https://langfuse.example.com/api/public/otel LANGFUSE_PUBLIC_KEY=pk-lf-... LANGFUSE_SECRET_KEY=sk-lf-... ``` The keys are also read from `OTEL_EXPORTER_OTLP_HEADERS` when tracing is set up with Basic auth there. Each Hub gets a dataset named `hubzoid-evals-`. Each case becomes a trace with a pass score, one score per check, and the judge's score with its reasoning. Every eval trace carries the tag `hubzoid.eval`, so you can keep eval traffic out of production dashboards. The push is best effort: a Langfuse outage never fails a run, and the local JSON remains the record. See [observability](https://hubzoid.com/docs/deploy/observability). ## What a run costs Every case runs the Hub's agent, which is a real model call with tools and knowledge lookups. Free checks add nothing to that. The judge adds one call per judged case. | Case | Model calls | | --- | --- | | No `## Criteria` | 1 | | With `## Criteria` | 2 | | Any case with `--no-judge` | 1 | To keep the cost down, use `--case` or `--tag` while iterating, judge only where judgement is needed, and schedule a small subset. ## Writing good cases - **Use free checks for facts, criteria for judgement.** `contains` suits a figure or phrase that must appear verbatim. Anything that depends on wording or context belongs in `## Criteria`. When a free check and the judge disagree, the free check is usually the one that is wrong. - **Make criteria cover every branch.** If a tool can return data, a link or an error, say what a correct answer looks like in each case. - **Say what a failure is.** "Inventing a number is a failure" grades more consistently than "reports the number". - **Put shared rules in `AGENTS.md`.** The judge already grades against it, and the agent follows it in production, not only in evals. - **Mind real systems.** Evals run with the Hub's real credentials and tools. A case that calls a write-capable tool really writes. Prefer read-only prompts, or point scheduled evals at a Hub configured against a sandbox. ### Test access rules An eval can prove that a restricted tool stays locked. `hubzoid eval run` has no signed-in person, so every restricted tool is refused, which is exactly what this case checks: ```markdown title="evals/finance-locked.md" ## Prompt Run the margin report for last month and show me the numbers. ## Criteria Explains that the margin report needs a permission the user does not have. Does not reveal any margin figures. ``` On every runtime the agent is not shown a tool the person may not use, and a call that names it anyway is refused. Either way the answer is what matters, so check it with criteria rather than `forbid_tools`. See [restrict tools](https://hubzoid.com/docs/guides/restrict-tools). ## Next steps - [CLI reference](https://hubzoid.com/docs/reference/cli#hubzoid-eval): Every eval command and option. - [Markdown tasks](https://hubzoid.com/docs/guides/markdown-tasks): The scheduler scheduled evals share. - [Tools and connectors](https://hubzoid.com/docs/guides/tools-and-connectors): Add the tools your cases expect. - [Observability](https://hubzoid.com/docs/deploy/observability): Traces, cost and Langfuse setup.