Operating runs

Markdown tasks and code workflows share one run history and one set of controls. The controls are hubzoid schedule commands on the server, and every pause, resume and cancel is recorded in the access audit. The Console shows runs, schedules, health and that audit.

When to use thisCopy Anchor Link

Use this page when you operate a hub that runs scheduled work: to see what ran, start something now, stop a schedule during an incident, cancel a stuck run, prepare a deploy or a backup, or work out why a run did not happen.

CommandsCopy Anchor Link

list and status default to the current folder. <name> is a markdown task name, with or without the md: prefix, or a code workflow's function name, where hyphens are read as underscores. Markdown task names are matched first. Run ids come from hubzoid schedule status or the Console. The full option list is in the CLI reference.

Run nowCopy Anchor Link

hubzoid schedule run queues the run on the same queue a scheduled run would use, so it never overlaps one, and it ignores the schedule, pauses and the chat gate. For a markdown task it accepts --timeout, --max-rounds, --model and --dry-run, as described in Markdown tasks. For a code workflow, --dry-run names the file it would run without executing any code, and the other three options are refused because those limits live in the workflow code. The command prints the result and exits 0 on success, 1 when the run failed or ended incomplete, and 2 when the name is unknown or the definition does not load.

Pause and resumeCopy Anchor Link

A pause is stored in the operational database, so every bridge on a shared database sees it. The scheduler reads pauses again right before it queues each run, so a pause that lands mid-tick still stops the next dispatch.

• Runs already queued or running continue. Cancel them if they must stop.
• Manual runs of a paused task or workflow still work.
• On resume, a markdown task that became due while paused runs once, the same catch-up as after downtime. A code workflow runs at its next slot without backfill.

CancelCopy Anchor Link

hubzoid schedule cancel my-hub md:weekly-count-summary:20260928T0717

Only a run that is queued or running can be cancelled. For any other run the command reports its status and does nothing. A queued run is cancelled before it starts. A running run stops at its next step boundary. Work already done is not undone: a message that was sent, a git push or a script's effects stay.

Audit attributionCopy Anchor Link

Each pause, resume and cancel is written to the access audit, with the operator recorded as cli:<user>@<host>, the account that ran the command on the server. The Console's Activity page shows these entries, for example "cli:ops@hub-01 paused the weekly-count-summary schedule in my-hub". Run controls act with the authority of whoever can run commands on the server. The Console shows paused work and the audit, and has no run buttons.

Read runs in the ConsoleCopy Anchor Link

Runs and schedules live inside each agent. On the agent's Runs & schedules tab, open a workflow with View runs to list its runs, newest first, with status, start time, duration, Runs as and result. Existing links to the cross-agent list from earlier releases still open Runs across your agents, which filters by agent, status, time range (24 hours, 7 days or 30 days) or run id, reloads every 10 seconds with auto-refresh, and keeps its filters in the URL.

Open a run to see its id, start and completion times, duration, the account it ran as, its result or the error that failed it, and each step with its own output or error. The first step records the account. A markdown run then shows its work step, its commit and push steps when they ran, and its finish step. A code workflow run shows your @step functions, model calls, publishes and emails.

A run's result belongs to the account it acted as. The hub's managers see its workflow, status, timing and a failure summary, while the result, step outputs and detailed errors read as private to that account. A legacy service run, which acts for no person, stays fully visible to managers. hubzoid schedule status on the server shows everything.

The Running status filter covers both queued and running runs.

Each agent's Runs & schedules tab lists its workflows and markdown tasks (as md:<task>) with schedule, timezone, state, Runs as and next run, and a line with the last dispatch and missed runs. Runs as names the account and where it came from, or says Cannot run with the fix.

Banners on the tab flag a stopped scheduler, slots missed while the scheduler was down and a workflow that could not load. See Console runs for the full screens.

Missed slotsCopy Anchor Link

• Markdown tasks catch up once after downtime or a pause. The catch-up run records how many extra slots it stood in for in the task's missed_log in .hubzoid/schedule-state.json, kept for 31 days.
• Code workflows never backfill. When the dispatcher falls behind, or a backup hold ends, it runs only the latest due slot and counts the earlier ones as missed. When the hub starts more than 90 seconds after the last heartbeat, it counts the slots that fell in the gap, logs a warning and shows a banner in the Console. Dated records are kept for 31 days.
• Each agent's Runs & schedules tab shows missed code workflow runs beside the last dispatch, and hubzoid schedule status prints a missed count per workflow. The Console's summary API (/portal/api/summary) adds both kinds into a missed-slots figure per agent for the chosen period.

Concurrency limitsCopy Anchor Link

Markdown tasks run one at a time per hub. Each code workflow runs one at a time, and different workflows run side by side. To cap how many code workflows run at once in a hub, set max_concurrent_workflows: N in workflows/settings.yaml and restart the hub.

A code workflow slot that comes due while the previous run is still going waits in the queue and runs after it. If a workflow regularly takes longer than its interval, runs pile up behind each other, so choose an interval longer than a typical run.

Drain before a deployCopy Anchor Link

Runs belong to the code that started them: a hash of the installed Hubzoid version and every workflows/**/*.py file. When the hub starts under a different version, runs that were queued or interrupted under the old code are cancelled so they cannot block a queue, and show as CANCELLED. Markdown runs that were queued but not started are queued again under the new code, with :requeued added to the id.

At shutdown, runs in flight get up to 5 seconds to finish. Anything still running stays recoverable and resumes at the next start if the code has not changed. To avoid cancelled runs, drain first:

Backup holdsCopy Anchor Link

hubzoid backup holds new scheduled runs for every hub in the deployment while it copies, and chat keeps working. It waits for running runs to finish first, for up to --wait seconds (default 600). If runs are still going when the wait ends, the backup stops with Scheduled runs are still in progress and names them. Try again later, cancel them, or pass --wait 0 to take the backup anyway.

When the backup ends, the hold is released. Due markdown tasks then run once, and each code workflow runs the latest slot that fell inside the hold. A hold also expires on its own if the backup process dies. While a hold is active, hubzoid doctor warns in scheduler.health. See Backup and restore.

Health checksCopy Anchor Link

hubzoid doctor <hub> reads without changing anything. Add --json for stable check ids.

See the doctor reference for every check.

TroubleshootingCopy Anchor Link

Next stepsCopy Anchor Link

Read this page as Markdown