Observability

Observability in Hubzoid is the record a deployment keeps of itself: usage for every model call, every restricted tool decision, every access change and sign-in, every run, and every webhook event and alert. Settings and the command line read those records directly from the deployment's own databases, while alerts and traces to an outside backend are optional and off until you set them up.

When to use itCopy Anchor Link

Use this page to set up monitoring for a deployment, to answer "who used what and what did it cost", and to investigate a refused tool call or a failed run.

The smallest real exampleCopy Anchor Link

hubzoid doctor reads a Hub and its deployment without changing anything. These two lines are from a starter Hub:

✓ Last backup 0.0 days ago. Scheduled backups run weekly. (backup.age)
✓ Scheduled work is not held or paused (scheduler.health)

Each check has a stable id in brackets. Run it with --json from a monitoring job and alert on the ids that matter to you.

How it worksCopy Anchor Link

Everything below lives in the deployment's own databases. Run history stays in the workflow engine's database, and Hubzoid keeps no second copy.

Webhook event and alert records have no automatic retention yet.

Usage and costCopy Anchor Link

Settings → Usage shows five totals for the last 24 hours, 7 days or 30 days, and this month's use by agent with its spending limit.

A figure Hubzoid cannot know reads Unavailable or a dash, never zero. The Agents list also shows each agent's chats and spend, and agent administrators see only their own agents.

Tool decisionsCopy Anchor Link

Every call to a restricted tool is checked in code and written as one row, allowed or denied. A call whose row cannot be written is refused, so every call that ran is in the log.

hubzoid audit ./finance --denied

Each line shows the time, the user, ALLOW or DENY, the tool, and the surface with a reason.

Activity → Audit log shows the same decisions under Tool decisions, and every grant, revoke, account change and sign-in event under Access changes, with filters for agent, time, person and action. Filters live in the URL, so a filtered view can be shared. See people and activity.

Scheduled runsCopy Anchor Link

Activity → Runs lists every run with its status, how it started and how long it took. Each agent's Workflows tab shows its workflows, whether each is on, who it runs as, and a link to its runs. A run's result and step outputs are shown only to the account the run acted as, and other managers see a failure summary. On the server:

hubzoid schedule status ./finance          # definitions, recent runs and errors
hubzoid schedule deliveries ./finance      # webhook events and alert deliveries

hubzoid doctor reports scheduler.health: a hold set by a backup in progress, paused work, a failed dispatcher, or no recent dispatcher heartbeat. See operating runs.

Workflow alertsCopy Anchor Link

Hubzoid can tell you when scheduled work goes wrong: failed runs (for webhook runs, only after their retries run out), failed scheduled eval suites, overdue runs, several failures in a row, a schedule that stopped firing, webhook events whose content changed under the same key, and a workflow engine that stopped responding. Repeats within an hour are combined into one alert with a count.

alerts:
  to:
    - webhook: ALERT_WEBHOOK_URL # the name of a variable that holds the URL
    - slack: SLACK_ALERT_WEBHOOK # a Slack incoming webhook URL, also by variable name
    - email: ops@example.com
  cooldown: 1h
  failures_in_a_row: 3
  pause_after_failures: 20

Health checksCopy Anchor Link

Watch /healthz/workflows from outside the machine, and supervise the processes as well, because an edge cannot report the death of its own host. The edge also sends an alert when a Hub's workflow engine goes stale and when it recovers.

Doctor reads only. It creates no database, runs no migration and starts no engine, so it is safe against a running deployment. It exits with status 1 when any check fails.

hubzoid doctor /opt/hubzoid/agents/finance --json

{
  "format": 1,
  "hub": "/opt/hubzoid/agents/finance",
  "hubzoid": "<version>",
  "ok": true,
  "checks": [
    {
      "id": "auth.bridge_keys",
      "status": "ok",
      "summary": "1 bridge key(s) set",
      "detail": null
    },
    {
      "id": "backup.age",
      "status": "warn",
      "summary": "No backup recorded. Scheduled backups run weekly. Run `hubzoid backup` now.",
      "detail": null
    }
  ]
}

Every check has a stable id and a status of ok, info, warn or fail. New checks may be added, and existing ids are never renamed.

OpenTelemetry tracesCopy Anchor Link

Hubzoid can push traces over OTLP/HTTP to any OpenTelemetry backend, such as Langfuse or an OpenTelemetry Collector. It is off until you set an endpoint in the Hub's environment:

HUBZOID_OTEL_ENDPOINT=https://langfuse.internal/api/public/otel
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64 of public_key:secret_key>

OTEL_EXPORTER_OTLP_HEADERS is the standard exporter variable and passes straight through. Emission is an outbound push, and nothing is added to the public port, so keep the receiver on a private network. When several Hubs run together, set HUBZOID_OTEL_ENDPOINT in each Hub's .env, pointing at the same backend. Traces separate by hubzoid.hub.

The spans come from the model SDKs, and what they carry depends on the model path.

LogsCopy Anchor Link

HUB_LOG_LEVEL sets the level for the bridge and edge: info (default), debug or warning. Hubzoid removes one-time sign-in link tokens and OAuth codes from its own access logs, but a reverse proxy in front of it still records them unless you redact them there.

RelatedCopy Anchor Link