Deploy and operate

Upgrading

Move a Hubzoid deployment to a new release with a verified backup, a short maintenance window and a tested way back if something goes wrong.

At the end of this guide your deployment runs the new release, every Hub has been checked, and you hold a backup that takes you back if you need it. Hub files rarely change between releases. What changes is the software around them: Hubzoid's own tables, the workflow engine, the web app and Settings. Each upgrades itself at the first start of the new version, so an upgrade is a short maintenance window with a backup taken just before that start.

What you need

Upgrade a deployment

  1. Read the notes
  2. Let runs finish
  3. Stop
  4. Back up
  5. Install
  6. Check
  7. Start
  8. Verify
Nothing is upgraded until the first start, so the backup is the deployment exactly as the old version left it.

Read the release notes

Note anything that needs a setting, a restart order or a grant, and whether the release changes the database schema. Rehearse on a copy of the deployment when a release changes how people sign in or how access works. What changed recently below covers the most recent changes.

Let scheduled work finish

Runs belong to the code that started them. After an upgrade, runs that were queued or interrupted under the old version are cancelled at the next start, so they cannot block the queue, and show as CANCELLED. Wait until no run is PENDING or ENQUEUED, ideally between scheduled slots:

Terminal
hubzoid schedule status ./my-hub

Stop everything

Stop hubzoid run, or its unit:

Terminal
sudo systemctl stop hubzoid

That stops every Hub of the deployment and their channels. Upgrade the whole deployment together. Running an old and a new version against the same databases is not supported.

Back up

Terminal
hubzoid backup ./my-hub --out pre-upgrade.tar.gz

A backup never creates or upgrades a table, so this archive is the deployment exactly as the old version left it. It includes a pg_dump of each PostgreSQL database. Keep protected copies of each .env and of the deployment key, which the archive leaves out. To be sure it restores, rehearse it with hubzoid restore pre-upgrade.tar.gz --rehearse. See backup and restore.

Install the new release

Set the pin in requirements.txt to the new release, then reinstall in the same environment. Keep any extras you use, such as hubzoid[postgres]==<version>.

Terminal
pip install -r requirements.txt

hubzoid version prints the installed release.

Check

Terminal
hubzoid doctor ./my-hub

Fix every check marked ✗. Before the first start, db.operational and db.hub report that their schemas are not upgraded yet. That is expected.

Start

Terminal
sudo systemctl start hubzoid

At the first start, Hubzoid and the workflow engine upgrade their own tables, then each Hub prints its ready line.

Verify

Sign in as an ordinary user and chat with each agent, use a tool that needs a grant, and download a file an agent makes. Open Settings: Agents, each agent's Workflows tab, and Activity. Run hubzoid doctor again. If a cancelled scheduled run is still needed, start it with hubzoid schedule run <hub> <name>.

What changed recently

Templates moved out of the package

Hubzoid no longer bundles Hub templates. hubzoid init <name> writes a small starter: every folder Hubzoid reads, three house rules for Kestrel & Oak and one check. Complete agents are ready-to-use agents in their own repositories: Software Factory, Order & Invoice Desk, Customer Desk and Engineering Desk.

  • Hubs you already made from a template keep working. hubzoid init copied the files into your Hub folder, and upgrading never deletes, regenerates or updates them.
Scripts, retired templates and the Docker demo folder
  • Scripts that call hubzoid init <name> --template <template> now stop with exit code 2, say where that template went, and create nothing. Remove the flag to get the starter, or clone a ready-to-use Hub.
  • The gitzoid template is now its own agent, Engineering Desk: git clone https://github.com/hubzoid/engineering-desk-hub engineering-desk. The operations, minimal, demo and watchtower templates are retired. Their files stay in the Hubzoid repository at the last release that bundled them, as the SDK upgrade guide (opens in a new tab) explains.
  • To start a new Hub from a retired template anyway, copy its folder from the repository, including hidden files such as connectors/.mcp.json, then add your own .env with MODEL= and a random BRIDGE_API_KEYS=. Do not run hubzoid init --force on an existing Hub for this, because it overwrites the starter's files in that folder.
  • The repository's demo-hub/ folder is gone, and docker/docker-compose.yml now needs HUB_PATH set to your Hub folder. See Docker.

Changing agent files from the browser is off by default

Proposing and approving Hub changes, the agent's remember and propose_hub_change tools, and saving connectors, channels, settings, approval rules and budgets in Settings now answer that this is off. People still read everything in Settings, which says where to change each item, and the command line and your editor work as before. To turn it back on, set in_app_editing: true in hubzoid.yaml (or HUBZOID_IN_APP_EDITING=true) and restart hubzoid run. Settings then marks those controls Beta. Files that come from an image are never written.

Settings has two groups

Settings is now Personal (Account, Connected accounts, Sources, Memory, My schedules) and Manage (Agents, Activity, Team schedules, People, Usage, Organization).

Where old pages moved, and each agent's tabs

Old addresses and bookmarks open the new place: Health is under Organization, runs and the audit log are under Activity, and My work items are in the Inbox. Each agent has the tabs Instructions, Access, Connectors, Workflows, Evals, Feedback and Configuration, plus Work when it has work items, incoming messages or sandboxes.

Other changes from that release

The release that turned off browser editing and regrouped Settings also brought eight more changes.

Every other change in that release
  • Work items on a Hub whose folder name has capitals, such as Customer-Desk, were queued where no worker listened. After the upgrade they run, including ones already waiting.
  • MODEL=codex-local works with Codex CLI 0.163.0 as well as 0.147.0. Other versions are refused with the versions to install. HUBZOID_CODEX_BIN names a binary that is not first on PATH.
  • On a machine with code workflow schedules off, the start warns when an agent has scheduled workflows, missed runs are no longer caught up, and Settings says schedules are off instead of showing a next run.
  • The API reference at a running Hub's /docs and /openapi.json is titled Hubzoid API and lists only the public API.
  • The chat's model picker shows only the agent's own model unless the deployment lists models under chat.models.catalog.
  • hubzoid run no longer stops when port 8000 is taken, because bridge ports skip busy ones, and its ready line says whether sign-in is on.
  • hubzoid audit --json and hubzoid test --json print only JSON.
  • Chat inside your own app is new: register an app under apps: in hubzoid.yaml.

That release changed no database tables or Hub files. Installing it and restarting, with hubzoid restart <hub> or the service, is enough.

Upgrading a gateway

One hubzoid backup of any Hub of a deployment covers its data directory and every Hub. Stop hubzoid run before installing, and every Hub starts again on the same release.

A deployment still run by hubzoid gateway, with separate units per Hub, moves to one hubzoid run once, offline:

  1. Let waiting work finish, then stop and disable the old gateway unit, every per-Hub unit and every Slack or inbound unit with sudo systemctl disable --now <unit>. Then back up each Hub and the gateway's data folder, and hubzoid backup verify the archive.

  2. Install the new release.

  3. In the folder that holds the Hub folders, see what would be written, then write it:

    Terminal
    hubzoid upgrade --dry-run --data-dir <old data folder>
    hubzoid upgrade --data-dir <old data folder>

    It writes hubzoid.yaml from the gateway's manifest, and moves settings kept only in the database (brand, chat defaults, diagnostics limits, connectors, keys typed into Settings, channels) into their files. It never overwrites a file and never prints a secret.

  4. Review the written files with git diff and commit them. Remove from each Hub's .env what is now set once, such as HUBZOID_AUTH (now sign_in:), and move sign-in client settings to the .env next to hubzoid.yaml.

  5. Write the one unit for the whole deployment, start it and check every Hub:

    Terminal
    sudo /opt/hubzoid/agents/.venv/bin/hubzoid service install --data-dir <old data folder> --user hubzoid --write
    sudo systemctl daemon-reload
    sudo systemctl enable --now hubzoid
    hubzoid status

To go back, follow going back and enable the old units again.

Upgrading a container

Back up, stop the container, build or pull the new image, and start again. With the Compose files:

Terminal
docker compose -f docker/docker-compose.yml exec hubzoid hubzoid backup /hub --out /hub/pre-upgrade.tar.gz
docker compose -f docker/docker-compose.yml down
git checkout v<version>
HUB_PATH=$HOME/my-hub docker compose -f docker/docker-compose.yml up -d --build

To run a published image instead, set HUBZOID_IMAGE=ghcr.io/hubzoid/hubzoid:<version>. The container upgrades its tables at its first start, the same as a native install. See Docker.

How database upgrades work

Hubzoid versions its own tables with Alembic in two stores, each with its own version table, so they never collide with each other or with the workflow engine's.

The two version tables, and how bridges take turns
StoreVersion tableHolds
Operationalhz_alembic_operationalAccounts, sessions, conversations, connectors, access, identities, audit, workflow catalog and state, usage, tool decisions
Hubhz_alembic_hubWhatsApp and Telegram conversation history

Each bridge upgrades both stores before it serves anything. Bridges that start together take a lock first (a PostgreSQL advisory lock, or a file lock next to a SQLite database), so only one runs a migration at a time. A table that does not match what Hubzoid expects is refused rather than silently stamped.

A database already upgraded by a newer Hubzoid is refused, and the bridge stops with a message naming the store. hubzoid doctor reports it as a failure on db.operational or db.hub. The fix is to install that newer version or restore a backup, which is why going back always means restoring the backup taken before the upgrade.

Going back

Stop everything. Restore with the new version still installed, then reinstall the previous release and start:

Terminal
hubzoid restore pre-upgrade.tar.gz
pip install "hubzoid==<previous version>"

Whatever the restore replaces is kept beside it as <name>.pre-restore-<time>. Add --postgres to load an archive's single PostgreSQL dump, or load several dumps with pg_restore first. Put back the matching .env files too, if you changed any for the new release. Backup and restore covers a restore that refuses a database still in use.

Deployments moving from a release that used Open WebUI follow the dedicated steps in the SDK's upgrade guide (opens in a new tab), including hubzoid migrate openwebui.

If something goes wrong

What you seeWhat to do
hubzoid init <name> --template <template> exits with code 2Templates left the package. Remove --template, or clone a ready-to-use Hub.
Set HUB_PATH to your hub folder, an absolute path from Docker ComposePass HUB_PATH with your Hub folder.
Settings shows connectors or channels read-onlyChanging agent files from the browser is off by default. Edit the files, or set in_app_editing: true in hubzoid.yaml.
A bridge stops and names a store upgraded by a newer HubzoidInstall that newer release, or restore the pre-upgrade backup.
A scheduled run shows CANCELLED after the upgradeIt was queued under the old version. Start it again with hubzoid schedule run <hub> <name>.
The start stops and names an Open WebUI settingRename or remove it as the message says. See the SDK's upgrade guide.

Next steps

Choose which cookies Hubzoid can use. You can change this at any time from Cookie settings in the footer. Read the Cookie Policy for details.