Backup and restore
Save a whole deployment to one verified archive while chat keeps working, rehearse a restore safely, and put it back on the same machine or a new one.
At the end of this guide you have a verified archive of your deployment copied off the machine, you have rehearsed restoring it without touching anything, and you know how to restore it for real. hubzoid backup saves databases, conversations, chat files and runtime state while people keep chatting. Every deployment also takes a verified backup by itself once a week.
Hub content (AGENTS.md, skills, knowledge, tools) belongs in the Hub's git repository and is not part of the archive.
What you need
- Shell access to the machine that runs the deployment, in the environment where Hubzoid is installed.
- For PostgreSQL, the PostgreSQL client tools (
pg_dump,pg_restore) at the server's major version or newer. The Hubzoid image includes them. - Somewhere off the machine to keep archives, and a separate safe place for the deployment key.
Back up and check
Take a backup
hubzoid backup ./my-hub --out /var/backups/hubzoid-$(date +%F).tar.gzFor several Hubs run together, back up any one of them: the archive covers the whole deployment. The command holds new scheduled runs, waits for running ones, writes the archive and reads it back before it counts:
Backup written and verified: /var/backups/hubzoid-2026-10-25.tar.gz (<n> files, <n> databases)
Left out: .env files, deployment encryption keys, signing keys and database passwords. Keep protected copies separately (see docs/BACKUP.md).Keep the deployment key apart
The archive leaves out the deployment key: secret.key in the deployment's data directory, <hub>/.hubzoid/secret.key for a standalone Hub, or the value of HUBZOID_SECRET_KEY. Copy it, and each .env, to a separate safe place once.
Without the key, restored connector credentials (people's connections, Shared keys and connector client secrets) and saved secrets cannot be decrypted, so people connect again. The rest of the data is unaffected.
Copy the archive off the machine
A backup on the same disk does not survive losing that disk. Copy archives with the tool you already use, for example:
rclone copy /var/backups remote:hubzoid-backupsAfter copying, check the copy is intact:
hubzoid backup verify /var/backups/hubzoid-<date>.tar.gz✓ Verified /var/backups/hubzoid-<date>.tar.gz: <n> files, <n> databasesverify checks every file against its recorded SHA-256, runs an SQLite integrity check on each database and lists each PostgreSQL dump. It exits with status 1 when anything fails.
Rehearse a restore
hubzoid restore /var/backups/hubzoid-<date>.tar.gz --rehearse--rehearse restores the archive into a scratch folder, upgrades each copy to the installed release, reports what came back (people, conversations, messages), then removes the scratch folder. It ends with ✓ Rehearsal passed and Nothing was changed. Rehearse before every upgrade, or once a quarter.
Scheduled backups
Every deployment backs itself up, weekly by default. The bridge looks every half hour and takes a verified backup when the last one is older than the schedule. A service that slept through its slot backs up soon after it wakes. In a deployment of several Hubs, the first Hub's bridge takes them.
| Setting | Default | What it does |
|---|---|---|
HUBZOID_BACKUPS | weekly | weekly, daily or off |
HUBZOID_BACKUP_KEEP | 4 | How many scheduled archives to keep. Older ones are removed. |
HUBZOID_BACKUP_DIR | .hubzoid/backups in the Hub or data folder | Where they go |
hubzoid backup list ./my-hub lists them, newest first. A failed scheduled backup is recorded and tried again six hours later. hubzoid doctor shows the last good backup as backup.age, and warns when there is none, the last one failed, or it is more than 7 days old. Scheduled archives still live on the same disk, so copy that folder off the machine too.
To use your own schedule instead, set HUBZOID_BACKUPS=off and run hubzoid backup from cron, for example daily with two weeks kept:
#!/bin/sh
set -e
cd /opt/hubzoid/agents
.venv/bin/hubzoid backup ./my-hub --out "/var/backups/hubzoid-$(date +%F).tar.gz"
find /var/backups -name 'hubzoid-*.tar.gz' -mtime +14 -deleteIn a container, run the backup inside it, for example docker compose -f docker/docker-compose.yml exec hubzoid hubzoid backup /hub --out /hub/backup.tar.gz.
What the archive holds
The archive holds every database, chat files and Hub runtime state. It leaves out Hub content, which belongs in git, and leaves out .env files and the deployment key unless you pass --include-secrets.
Everything saved and not saved
| Saved | From |
|---|---|
| Every SQLite database | The operational store (accounts, sessions, conversations, grants, access changes, usage, workflow state and encrypted connector credentials), each Hub's workflow engine database and the Hub database. Copied with SQLite's online backup, so each copy is consistent while the Hub runs. |
| PostgreSQL databases | A pg_dump of each distinct database named by DATABASE_URL, HUBZOID_OPERATIONAL_DB or HUBZOID_DBOS_DB. See PostgreSQL. |
| Chat files | Uploads and files the agent made, under each Hub's .hubzoid/chats/ |
| Hub runtime state | Each Hub's .hubzoid/, .inbound/, logs/ and output/ folders |
| Not saved | What to do |
|---|---|
| Hub content | Keep it in git. Check it out again before a restore to a new machine. |
.env files | Keep a copy of each somewhere safe, or pass --include-secrets. |
| The deployment key | Back it up apart from the data, or pass --include-secrets. |
.hubzoid/artifact_secret | Signs download links. Without it, links issued before the restore stop working. Saved with --include-secrets. |
Database passwords in deployment.json | Saved as ***, unless --include-secrets |
| Earlier backups | The scheduled backups folder is never inside the next archive. |
The archive is written with owner-only permissions. It still holds accounts, password hashes and conversations, so store it like a secret.
hubzoid backup options: --out (default ./hubzoid-backup-<time>.tar.gz, refuses a path that exists), --include-secrets, and --wait (seconds to wait for running scheduled work, default 600).
What happens to running work
A backup never interrupts chat, Slack, MCP or webhooks. Scheduled work is paused around the copy so that run history and output files match.
- Hold new runs
- Settle 5 seconds
- Wait for running work
- Write and verify archive
- Release hold
When runs are still going, and what a backup writes
- If runs are still going after
--waitseconds, the backup stops without an archive and names them. Try later, cancel them withhubzoid schedule cancel <hub> <run-id>, or pass--wait 0. - With
--wait 0the backup skips the settle and does not wait. A run caught part way is reported as interrupted after a restore and runs again at its next slot. - If the backup process dies, the hold expires by itself after the wait time plus three hours. While it is active,
hubzoid doctorreports it underscheduler.health.
Apart from its hold and the record of the backup, a backup only reads. It never creates or upgrades a table, so you can back up with a newer Hubzoid installed before its first start. See upgrading.
hubzoid restore
Stop hubzoid run, or its unit, first. Restore refuses to replace a database another process is using.
hubzoid restore backup.tar.gz --dry-run
hubzoid restore backup.tar.gzEvery restore option
Prop
Type
Each saved directory goes back to the path it came from. Whatever is at that path now is kept beside it as <name>.pre-restore-<time>, so you can undo a restore by moving those back. Delete them once the restored deployment checks out. Secret files the archive does not hold stay as they are at the target.
Archive checks, and databases that may be in use
Restore checks the whole archive before it changes anything. It refuses unsafe entries, a target where a backup never saves from (such as a home directory), and targets that overlap.
Restore also refuses a SQLite database that still has a lock or a -wal or -shm file beside it, because a process may be using it. After an unclean shutdown, stop every process that uses the database, keep the database with those files, and checkpoint it before you retry:
sqlite3 /path/to/database.db 'PRAGMA wal_checkpoint(TRUNCATE);'Do not delete those files to force a restore, because they can hold committed data. If you cannot confirm every writer has stopped, restore to a separate location with --move.
If the deployment manifest was saved with *** in place of passwords, restore says so. hubzoid run rewrites the manifest from the deployment's environment every time it starts. After a restore, start hubzoid run, run hubzoid doctor <hub>, and open Settings to check access and recent runs.
Restore to a new path or machine
- Install Hubzoid at the version that wrote the archive or a newer one. An older release refuses databases a newer one has upgraded.
- Clone the Hubs' git repositories at their new paths.
- Run
hubzoid restore backup.tar.gz --move /root/hubs=/srv/hubs. Restore rewrites the absolute paths Hubzoid stores: the deployment manifest, each Hub's pointer to it and the schedule state. - Copy each Hub's
.env, the deployment key and the download-link signing key into place, unless the archive was made with--include-secrets. - Start
hubzoid run, runhubzoid doctor, and sign in.
A hosted deployment should set HUBZOID_SECRET_KEY in the provider's secrets, so a new volume restored from an archive can still read stored connector credentials.
PostgreSQL
With DATABASE_URL, HUBZOID_OPERATIONAL_DB or HUBZOID_DBOS_DB on PostgreSQL, hubzoid backup runs pg_dump once per distinct database and puts each dump in the archive as postgres/<n>.dump. The password reaches pg_dump through its environment, not its command line. Set HUBZOID_PG_BIN to the folder of the client tools when they are not on PATH.
How PostgreSQL dumps line up with the files
The dumps are taken while new scheduled runs are held. Chat keeps working, so a message sent during the backup may be in a dump and not in the files, or the reverse. For an exact match, stop the deployment and run hubzoid backup --wait 0.
To restore an archive with one PostgreSQL database, stop the deployment and load the dump with the files. The URL is read from an environment variable, never the command line:
hubzoid restore hubzoid-<date>.tar.gz --postgrespg_restore replaces what that database holds in one transaction, so a failure leaves it as it was. An archive with several PostgreSQL databases, such as separate operational and workflow databases, is not loaded by --postgres. Extract each postgres/<n>.dump from the archive and load it into its database with pg_restore --clean --if-exists --no-owner --dbname <url>, then run hubzoid restore without --postgres for the files.
To rehearse an archive with a PostgreSQL dump, name a server where a temporary database can be created and dropped:
export REHEARSAL_DATABASE_URL=postgresql://hubzoid@localhost/postgres
hubzoid restore hubzoid-<date>.tar.gz --rehearse --postgres-env REHEARSAL_DATABASE_URLIf something goes wrong
| What you see | What to do |
|---|---|
| The backup stops and names running scheduled runs | Wait and try again, cancel them with hubzoid schedule cancel, or pass --wait 0. |
backup.age warns in hubzoid doctor | No backup yet, the last one failed (doctor says why), or it is over 7 days old. Run hubzoid backup. |
✗ Not verified from hubzoid backup verify | The archive is damaged or incomplete. Use an earlier copy, and check how it was copied. |
... may be in use: a lock or SQLite journal remains. | Stop the deployment. If every writer has stopped, checkpoint the database with sqlite3 <database> 'PRAGMA wal_checkpoint(TRUNCATE);'. |
This archive holds 2 PostgreSQL databases. | Load each dump with pg_restore, as described under PostgreSQL. |
| Saved secrets show "Cannot be read: set again" after a restore | The deployment key was not restored. Put back the original key, or save those secrets again and have people reconnect. |
| Old download links stop working after a restore | The .hubzoid/artifact_secret was not restored. Put it back, or share new links. |