Operating runs

list, status and deliveries use the current folder when you leave out the Hub, and list, status and deliveries take --json. <name> is a markdown task name, with or without the md: prefix, or a code workflow's function name, where hyphens are read as underscores. Markdown task names are matched first. Run ids come from hubzoid schedule status or Settings.

For a markdown task it accepts --timeout, --max-rounds, --model and --dry-run, as described in markdown tasks. For a code workflow, --input key=value passes typed inputs, --dry-run names the file it would run without running any code, and the other three options are refused. The command prints the result and the links of any live reports the run published. It exits 0 on success, 1 when the run failed or ended incomplete, and 2 when the name is unknown, an input is invalid or the workflow does not load.

In Settings, Run now shows a form for the workflow's inputs first when it has any. If a run is already queued or running, Settings says so and starts nothing. If Settings says you can't run it, give yourself run access on the agent's Access tab, under Workflow permissions. Markdown tasks and code workflows both have Copy run command in the row's menu, which copies the server command. Webhook workflows have neither.

A pause is stored in the operational database, so every bridge on a shared database sees it, and it is checked again right before each run is queued.

• Runs already queued or running continue. Cancel them if they must stop.
• Manual runs of a paused task or workflow still work.
• On resume, a markdown task that became due while paused runs once. A code workflow runs at its next slot without catching up.
• A schedule whose scheduled runs fail 20 times in a row is paused automatically. See alerts.

For any other run the command reports its status and does nothing. A queued run is cancelled before it starts. A running run stops at its next step. Work already done is not undone: a sent message, a git push or a script's effects stay. Cancelling a webhook run fails its event, which then waits for an explicit redrive.

The tools act only on the agent they run in. A run started from chat acts as the workflow's own account, never as the person asking, and that person is recorded as the one who started it. The agent names the workflow and waits for a yes before it acts, but the grant, the check on every call and the audit are the real controls. The tools work in the web app, the API, MCP, WhatsApp and Telegram, and are never offered on Slack or inside a scheduled run. HUBZOID_WORKFLOW_TOOLS=false in a Hub's .env removes them from that agent.

A server command records the operator as cli:<user>@<host>. A start from chat or Run now records the person and where they acted. Server commands act with the authority of whoever can run them on the server.

A run's result belongs to the account it acted as. The Hub's managers see its workflow, status, timing and a failure summary, while the result, step outputs and detailed errors read as Private. hubzoid schedule status on the server shows everything.

A personal schedule pauses itself, with the reason shown, after three failures in a row, when the person's access changes or when the workflow's inputs change. Only an administrator resumes a schedule an administrator paused. People's schedules run only while the Hub runs with code workflows on, so a laptop that sleeps does not run them.

• Failed runs, overdue runs, failed eval suites and stopped schedules have a cooldown per workflow, one hour unless cooldown says otherwise, and the next alert counts what the cooldown held back.
• alert_to= on a code workflow, or alert_to: in a markdown task, replaces the Hub destinations for that workflow. alerts: {to: []} turns alerts off for a Hub. on_failure from older releases still works as one webhook destination. HUBZOID_ALERT_URL is the deployment's fallback webhook.
• An alert carries the Hub, its kind, a link to Settings and, when they apply, the workflow, the run id and a count. Run output and error text stay on the server.
• A webhook delivery is a JSON POST with Idempotency-Key and X-Hubzoid-Timestamp headers. With HUBZOID_ALERT_SECRET set, it also carries X-Hubzoid-Signature: sha256= and the hex HMAC-SHA256 of the timestamp, a . and the raw body. Email uses the deployment's SMTP settings, and preview mode writes to .hubzoid/alert-preview/.
• Delivery is at least once. A receiver can see a repeat after a network failure, so deduplicate on the idempotency key.
• A missing destination or a delivery that ran out of attempts shows in hubzoid schedule deliveries, never as a success.
• The first start of a Hub looks back 24 hours. Older failures are not alerted. After that, downtime of any length is caught up.

• Markdown tasks catch up once after downtime or a pause. The catch-up run records how many extra slots it stood in for in the task's missed_log in .hubzoid/schedule-state.json, kept for 31 days.
• Code workflows do not catch up. When the scheduler falls behind, or a backup hold ends, only the latest due slot runs and earlier ones count as missed. When the Hub starts more than 90 seconds after the last heartbeat, it counts the slots that fell in the gap, logs a warning and shows a banner in Settings. A workflow declared with catch_up=True runs the latest missed slot once, unless its next slot is close.

To cap how many code workflows run at once in a Hub, set max_concurrent_workflows: N in workflows/settings.yaml and restart. A workflow with its own concurrency=N or concurrency_key has a separate queue and limit, as described in webhooks.

A code workflow slot that comes due while the previous run is still going waits and runs after it. If a workflow often takes longer than its interval, runs pile up, so choose an interval longer than a typical run.

The code is a hash of the installed Hubzoid version and every workflows/**/*.py file. Markdown runs that were queued but not started are queued again, with :requeued added to the id. Webhook events that never started run on the new code, and an event whose attempt had started is failed for an explicit redrive.

At shutdown, runs in flight get up to 5 seconds to finish. Anything still running resumes at the next start if the code has not changed.

It first waits up to --wait seconds (default 600) for running runs to finish. If runs are still going, the backup stops with Scheduled runs are still in progress and names them. Try again later, cancel them, or pass --wait 0 to take the backup anyway.

When the backup ends, the hold is released. Due markdown tasks then run once, and each code workflow runs the latest slot that fell inside the hold. A hold also expires on its own if the backup process dies. While a hold is active, hubzoid doctor warns under scheduler.health, and a start from chat or Run now is refused.

For an external uptime monitor, the public port answers GET /healthz/workflows with 200 while every Hub's workflow engine is healthy and 503 otherwise. The answer names no Hub and no error. Monitor it from another machine, because a host cannot report its own failure, and supervise the process separately.