Observability

Hubzoid records what operators need in its own operational store: usage for every chat turn and workflow model call, every restricted tool decision, and every access change. The Console and the CLI read those tables directly, so the numbers are the same whichever chat app fronts the agents. Traces to an external backend are optional and off by default.

When to use thisCopy Anchor Link

Use this page to set up monitoring for a deployment, to answer "who used what and what did it cost", and to investigate a refused tool call or a failed run. For the Console screens themselves, see the Console overview.

What Hubzoid recordsCopy Anchor Link

All four live in the deployment's own databases. Hubzoid keeps no second copy of run history and sends none of this anywhere.

Usage and costCopy Anchor Link

The Console opens on Agents: five totals for the last 24 hours, 7 days or 30 days, then one card per agent with its tokens and approximate cost for the same period. Agent administrators see only their agents.

A figure Hubzoid cannot know reads Unavailable or a dash, never zero. Refused tool calls are listed under Activity as Tool decisions with the outcome Denied, and missed code workflow runs show on each agent's Runs & schedules tab. See the Console overview.

Tool decisionsCopy Anchor Link

Every call to a restricted tool is checked in code and written as one row, allowed or denied. A call whose row cannot be written is refused, so every call that ran is in the log. The chat app never sees tool calls, so this log is the only record of them.

hubzoid audit ./finance                    # the 50 most recent decisions
hubzoid audit ./finance --denied           # refusals only
hubzoid audit ./finance --user priya@example.com -n 200

Each line shows the time, user, ALLOW or DENY, the tool, and the surface with a reason:

The Console's Activity page shows the same decisions under Tool decisions, and every grant, revoke and account change under Access changes, with filters for time, agent, person, tool, outcome and channel. Filters live in the URL, so a filtered view can be shared. See people and activity.

Scheduled runsCopy Anchor Link

Each agent's Runs & schedules tab in the Console lists its workflows, who they run as, and every run with its status and steps. A run's result and step outputs are shown only to the account the run acted as, and other managers see a failure summary. The tab is read only. Control runs on the server:

hubzoid schedule status ./finance          # definitions, recent runs and errors
hubzoid schedule pause ./finance month-end
hubzoid schedule cancel ./finance <run-id>

hubzoid doctor reports scheduler.health: a hold set by a backup in progress, paused work, a failed dispatcher, or no recent dispatcher heartbeat. See operating runs.

OpenTelemetry tracesCopy Anchor Link

Hubzoid can push traces over OTLP/HTTP to any OpenTelemetry backend, such as Langfuse or an OpenTelemetry Collector. It is off until you set an endpoint in the hub's environment:

HUBZOID_OTEL_ENDPOINT=https://langfuse.internal/api/public/otel
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64 of public_key:secret_key>

OTEL_EXPORTER_OTLP_HEADERS is the standard exporter variable and passes straight through, for backends that need authentication. Emission is an outbound push. Nothing is added to the public port, so keep the receiver on a private network.

The spans come from the model SDKs. What they carry depends on the model path:

On the claude-local path Hubzoid sets these defaults for the Claude Code process, and a value you set yourself wins:

Claude Code puts token counts under names Langfuse does not map, so cost shows as zero there. Set HUBZOID_OTEL_NORMALIZE=true to fix this inside the bridge, with no extra process: the bridge receives its own Claude Code spans on its loopback port, copies the token counts to the gen_ai.usage.* names, sets each span's user.id to the signed-in person, adds the hub name as a Langfuse tag so cost can be grouped by hub, and forwards the batch to HUBZOID_OTEL_ENDPOINT. It has no effect on the LiteLLM path. If you already run a shared collector, point HUBZOID_OTEL_ENDPOINT at it instead, leave normalization off, and use the reference configuration in docs/otel-collector.yaml (opens in a new tab).

Langfuse prices tokens from its own model table. A model it does not know yet shows tokens with zero cost until you add its price there.

In a gateway every hub runs its own bridge, so set HUBZOID_OTEL_ENDPOINT in each hub's .env, pointing at the same backend. Traces separate by hubzoid.hub.

Two related switches are separate from HUBZOID_OTEL_ENDPOINT. HUBZOID_OPENAI_TRACING=true lets the OpenAI Agents SDK export its own traces, including prompts and tool data, to OpenAI. It is off by default. Eval runs push one trace per case with scores to Langfuse when LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY are set (or read back from the Basic credentials in OTEL_EXPORTER_OTLP_HEADERS), tagged hubzoid.eval so you can filter them out. See evals.

LogsCopy Anchor Link

HUB_LOG_LEVEL sets the level for the bridge and edge: info (default), debug or warning.

Health checksCopy Anchor Link

Doctor reads only. It creates no database, runs no migration and starts no engine, so it is safe against a running deployment. It exits with status 1 when any check fails.

hubzoid doctor /opt/hubzoid/agents/finance --json

{
  "format": 1,
  "hub": "/opt/hubzoid/agents/finance",
  "hubzoid": "1.0.1",
  "ok": true,
  "checks": [
    { "id": "auth.bridge_keys", "status": "ok", "summary": "1 bridge key(s) set", "detail": null },
    { "id": "backup.age", "status": "warn", "summary": "No backup recorded. Run `hubzoid backup`.", "detail": null }
  ]
}

Every check has a stable id and a status of ok, info, warn or fail. ok is false when any check fails. New checks may be added, and existing ids are never renamed, so alert on ids. The ones most useful for monitoring:

The doctor reference lists every check.

Next stepsCopy Anchor Link

Read this page as Markdown