Backup and restore¶
Create a backup snapshot to recover the vault independently of its live storage. A snapshot includes the folder tree, retained document versions, and their retained derivatives, whether stored individually or in a pack. It excludes unreferenced bytes waiting for garbage collection. Restore verifies the snapshot before publishing a usable vault at a separate target.
On 64-bit systems, backup and restore can read existing packed content larger than 512 MiB without rewriting the packs. Reading older compressed content can require memory proportional to the full uncompressed file size.
A repository is a directory of backup files that Docbank adds to without changing existing files. Each snapshot has a manifest, the record of which files belong to it. Repeated captures reuse unchanged content by hash.
Each snapshot also includes a complete metadata description in JSONL, with one JSON record per line. Docbank reuses this description when it is unchanged. A metadata change stores one new compressed description rather than a set of individual row changes. See Backup architecture for the repository format.
Repositories are not encrypted
Snapshot metadata and document content are compressed but not encrypted. Protect the repository with filesystem permissions and encrypted storage, especially before placing it on removable or cloud-synchronized media.
Use backup forget to remove selected recovery points and backup prune to reclaim their storage. Embedded applications can use the same cleanup operations.
Quick start¶
# One time
docbank backup init --repo ~/Backups/docbank
# Capture the live vault through its daemon
docbank backup create --repo ~/Backups/docbank --tag before-reorganization
# Inspect the snapshot history
docbank backup list --repo ~/Backups/docbank
# Prove the latest snapshot, including every referenced content byte
docbank backup verify --repo ~/Backups/docbank
# Recover and prove it without touching the running vault
docbank backup restore --repo ~/Backups/docbank --target ~/Restores/docbank-test
Set a default repository to omit --repo:
Restart the daemon after changing config.toml. Configuration is read only at
daemon startup.
Schedule backups¶
Initialize the repository, then configure a schedule in config.toml:
Restart the daemon to apply it. The daemon captures immediately when no scheduled snapshot exists or the newest one is due. Later captures run at the configured interval. An overdue capture starts within one minute after the computer wakes. An enabled schedule keeps the background daemon running through idle periods. The repository must already be initialized. Capture failures retry after the shorter of the interval and five minutes.
The schedule runs only while the daemon runs. After a reboot, the daemon starts
again when a Docbank command or the web app next needs it, and an overdue
capture begins then. To keep captures current across reboots, run
docbank daemon start from a login item or a launchd or systemd user service.
Operators can override the interval and retention without restarting:
docbank backup schedule set interval 1h
docbank backup schedule set keep_last 5
docbank backup schedule reset interval
docbank backup schedule show
show prints the overrides beside the interval and retention in force. Interval
changes apply immediately; keep_last applies at the next capture's retention.
Reset inherits config.toml. An in-flight capture finishes with the retention
count it started with. Overrides use the vault settings revision and travel with
backup and restore; a restored vault's schedule applies after
docbank backup claim. The repository path stays in config.toml. When the
effective schedule cannot run, for example because a restored vault carries an
interval override to a home without [backup] repo, Jobs reports the problem and
the daemon can still stop when idle.
After capture, retention keeps the newest keep_last scheduled points,
including the new one, and forgets older points. It prunes unused storage on
the first capture after the daemon starts and after it forgets a point. Manual snapshots
stay available. Cleanup failures leave the new point available and retry after
the next scheduled capture.
Scheduled points use docbank:scheduled:<vault-id>. Retention selects only this
vault's tag. Manual captures through the CLI, HTTP API, and embedded API reject
the reserved docbank:scheduled: prefix. The plain tag docbank:scheduled
remains an ordinary manual tag.
docbank jobs shows the schedule's latest error. When the newest scheduled
point is older than twice the interval, Jobs reports it as stale, and the daemon
logs one warning per stale episode. This check reads snapshot times after
restart. A future timestamp caps the initial wait at one interval; restarting
resets that wait. A backward clock correction cannot extend an active wait.
Set interval = "0" to disable scheduling. Enabling it requires repo, a
positive keep_last, and an interval of at least one minute.
After upgrading, claim existing history before resuming manual or scheduled captures.
Initialize a repository¶
Initialization creates the repository layout and its random identity, then assigns it to the current vault installation. Each repository belongs to one vault; captures from another vault are refused. Moving or upgrading a vault keeps its installation. Restoring a vault creates a new installation, so the restored copy must claim the repository before it captures. The source-independent embedded initializer leaves an empty repository unassigned until its first vault capture.
Initialization refuses an existing non-empty or already initialized destination.
An empty unmarked repository binds to this vault on its first capture. The
repository must be configured or supplied explicitly. A relative CLI --repo is resolved from the invoking shell's
working directory before it is sent to the daemon.
Adopt an existing repository¶
A repository needs an explicit claim before Docbank adds snapshots or uses its
history for scheduling when it has history but no docbank-owner marker, as
after upgrading from a release without ownership, or when another installation
of this vault owns it, as after a restore:
The command shows the repository identity, snapshot count, and newest snapshot
time. When another installation of this vault owns the repository, it says so:
claiming moves capture access to this installation, and the other installation's
captures then require a claim. Review the history, then answer y or yes in a
terminal to assign it to the selected vault. Use --yes for automation.
--json prints the final claim object with owned true or false; previews and
prompts go to stderr.
The confirmation applies only to the history you reviewed. If the repository
changes between the preview and the confirmation, the claim fails with
backup_claim_changed and exits 4; run the command again to review the new
history.
A claim preserves existing snapshots and cannot take a repository from another
vault. Scheduled captures report the claim requirement in Jobs and leave
repository history untouched. Captures into a repository another vault owns
return HTTP 409 with backup_repository_owner_conflict and exit 2 in the
CLI. History this installation has not claimed returns 409 with
backup_repository_claim_required and exits 7. Claims require the operator's
API credential. Embedded callers use
Vault.PreviewBackupClaim and Vault.ClaimBackupRepository.
Docbank releases without ownership can still list, verify, and restore owned repositories. They do not check the owner marker, so they can also add manual snapshots to them. Scheduled retention never removes those snapshots.
Create a snapshot¶
docbank backup create [--repo DIR] [--tag LABEL] [--jobs N]
[--force-unlock] [--progress auto|bar|plain] [--json]
The daemon captures one consistent view of the vault. It briefly pauses changes while opening a SQLite read transaction. Normal writes then resume into SQLite's write-ahead log (WAL), while backup reads the earlier view.
GC, trash empty, verification, and packed-storage maintenance wait until the capture finishes. This keeps them from removing content needed by the snapshot. The daemon accepts a document into the repository only after reading and verifying all of its bytes. If capture fails, it publishes no partial snapshot manifest. Address the error, then rerun the command.
During an interactive run, --progress auto draws an in-place bar for each
stage (freeze, metadata, attachments, and seal), including item and byte counts
when available. Redirected output uses throttled, newline-terminated progress
instead, so logs remain readable. Force either behavior with --progress bar
or --progress plain. Progress goes to stderr. --json suppresses it and
writes one snapshot object to stdout for automation.
--jobs 1 serializes blob readers for repositories on spinning disks or NAS
storage. Zero uses Kit's CPU-based default. --tag is a free-form label shown
by backup list. --force-unlock is recovery for a known-dead repository lock,
not a way to override another running backup.
List snapshots¶
The table reports each snapshot's ID, creation time, logical file/blob counts,
total blob bytes, bytes newly added to the repository, recorded coverage, and
tag. --json returns the same typed
snapshot summaries as GET /api/v1/backup/snapshots, including the metadata
format and parent snapshot ID.
What each snapshot contains¶
Create and list output include recorded coverage: photo assets and files, distinct verified photo bytes, albums and members, hidden and trashed photos, checkout definitions and entries, and retained change receipts. Browser sessions receive snapshot lists without coverage. JSON summaries expose these under coverage, with version: 1, included, and excluded; top-level blob_bytes records all verified blob bytes. List shows one row per snapshot and one shared exclusions line. Older snapshots show unrecorded in the coverage columns and omit the JSON field. Snapshots with a newer coverage version show unsupported and an ID-specific warning; JSON includes coverage_warning and omits coverage. Their base statistics and IDs remain available for cleanup. Restore recounts version-1 coverage before publication, preserves the recorded exclusions, and rejects unsupported versions with upgrade guidance. Verification checks manifest consistency and physical content, and reports unsupported coverage versions; it does not recount photo coverage.
Upgrade Docbank before restoring snapshots that record coverage version 1 with a binary predating that coverage format. Older binaries refuse publication with a statistics mismatch; the manifest’s physical reader version does not express this application requirement.
Coverage excludes browser sessions, hidden unlock sessions, transient storage operations, checkout destination bindings, and rebuildable search indexes, photo timelines, quality signals, photo_technical_metadata, vector indexes, and pending photo suggestion text. Suggestion dismissals and accepted-field decisions are backed up but not counted in coverage; restore describes the undecided fields again after fresh consent. Hidden passcodes and lockout decisions remain included. Ingest provenance, saved-query runs, and term-report history remain included too. The trashed-photo count includes any asset with at least one trashed managed file, even when another member remains live. Restored checkouts are unbound until you explicitly rebind them.
Remove recovery points and reclaim storage¶
docbank backup forget SNAPSHOT [SNAPSHOT...] [--repo DIR] [--dry-run] [--allow-empty] [--force-unlock] [--json]
docbank backup prune [--repo DIR] [--dry-run] [--force-unlock] [--json]
Use exact IDs from backup list; unknown or misspelled IDs return 404. Forget refuses the last recovery point unless you pass --allow-empty, and refuses a parent required by a retained incremental snapshot. Forget removes manifests; prune separately removes unused packs and rewrites packs with less than half their encoded content still live. Dry runs take the same repository lock and report the planned work. Text output starts with dry run: nothing removed.
Ownership checks govern captures and automatic retention. Explicit forget and
prune operate on the repository selected by the operator, independently of its
source vault.
These commands require the master daemon credential. Browser sessions cannot run them. Cleanup errors with selected, planned, or completed work include partial reports in the API problem's backup_forget or backup_prune field. The CLI prints the report in text or JSON before returning the cleanup error and exiting nonzero. Text lists selected and forgotten snapshot IDs, planned pack removals and rewrites, and removed pack names. Items absent from completed lists remain pending or unconfirmed; bytes written alone do not prove a rewrite completed. Planned bytes are not net savings, and cleanup does not securely erase content.
Recover on a fresh installation¶
Use a temporary fresh home to run the recovery daemon, and restore into a separate directory. The target must be disjoint from that home and the backup repository.
DOCBANK_HOME=~/Recovery/control docbank daemon start
DOCBANK_HOME=~/Recovery/control docbank backup restore --repo ~/Backups/docbank --target ~/Recovery/vault
DOCBANK_HOME=~/Recovery/control docbank daemon stop
DOCBANK_HOME=~/Recovery/vault docbank vault info
Restore validates photo relationships and recorded coverage before publishing the database, verifies managed content, clears sessions, and leaves checkouts unbound. Keep the backup repository available until you've checked the restored vault.
Verify repository integrity¶
docbank backup verify [<snapshot>] [--repo DIR] [--all] [--quick] [--jobs N]
[--force-unlock] [--progress auto|bar|plain] [--json]
With no snapshot argument, Docbank verifies the latest snapshot. Pass a
snapshot ID to check a historical snapshot, or --all to check every manifest.
A full verification performs these checks:
- Read the repository indexes and pack footers.
- Reconstruct the snapshot's JSONL metadata.
- Read every referenced content object and verify its SHA-256 hash.
- Check recorded totals against manifest attachment totals and report unsupported coverage versions. Restore separately recounts version-1 coverage from the staged database before publication.
Content shared by several selected snapshots is read only once during one verification pass. Every finding is reported with the affected snapshot, and the command exits non-zero if any problems were found.
--quick checks manifest statistics and coverage versions, indexes, pack structure, metadata, and logical
references without reading document content. Its bytes_read therefore still
includes metadata bytes. It is useful after each capture, but it does not prove
the storage medium still holds every content byte. Run full verification
regularly. --jobs 1 avoids concurrent reads on spinning disks and
latency-sensitive network storage. The progress and JSON contracts match
backup create: progress is written to stderr, and --json suppresses
progress so stdout contains one typed report.
Snapshots include the folder tree, trash state, retained versions, ingest and provenance records, watched-source cursors, tags, saved queries, and extraction records. They also include source metadata, renditions, visual previews, and stored embedding results. Restore rebuilds and validates these records without calling processing providers. See Backup architecture for retained records and rebuildable indexes.
Snapshots keep photo suggestion dismissals and accepted-field decisions, plus a marker for an unconfirmed rotation proposal. They omit generated text and rotation angles. After restore and fresh processing consent, only undecided fields are described again. A dismissed caption stays dismissed while its pending rotation regenerates; a confirmed rotation needs no regeneration.
Restore and prove a snapshot¶
docbank backup restore [<snapshot>] --target DIR [--repo DIR] [--overwrite]
[--jobs N] [--force-unlock]
[--store-map OWNER_PRIVATE_FILE]
[--progress auto|bar|plain] [--json]
Restore selects the latest snapshot by default. Pass a snapshot ID to recover a
historical point. The CLI resolves --target from its working directory before
sending an absolute server path to the daemon. The target must be separate from
both the running vault and the repository. Direct paths, parents, descendants,
and symlink aliases that overlap either one are rejected. Filesystem-identity
checks also reject differently cased or Unicode-normalized spellings that
identify the same tree on filesystems where those names are equivalent.
How does restore protect the target?¶
Every restore pins the target directory and takes its vault-tree lock before
writing, including for a fresh or empty target. That excludes a second restore,
a restore to any ancestor or descendant, and a daemon rooted anywhere in the
same tree. Replacing the target pathname while restore is running cannot
redirect publication. A successful restore leaves the ordinary vault.lock as
part of the usable vault. A failed restore also leaves that advisory file in
place after releasing it. Retries ignore vault.lock when deciding whether the
target contains payload, and keeping the pathname avoids split-lock races
between old and newly created lock files.
A new or empty target needs no destructive flag. A non-empty target is refused
unless you pass --overwrite. Overwrite is a merge: files absent from the
snapshot remain in place. The old database and SQLite sidecars remain intact
until all repository content has been read and verified, the replacement
database passes integrity_check, and its logical statistics match the
manifest. Only then is the database published. Failure or cancellation before
publication leaves an existing database intact and creates no docbank.db for
a new target. If cleanup fails after publication, restore reports an error and
keeps the new database and its ownership marker. Earlier failures restore the
prior marker. An interrupted handoff is
reconciled when the vault is opened or the restore is retried, against a
durable fingerprint of the prior database and the validated identity of
whichever database was published. An unrelated file named docbank.db is never
opened for mutation to decide which side won.
Where does restored content go?¶
By default, restore copies compatible repository packs, verifies them, writes them durably, and records them as readable locations in the catalog. If a pack or object exceeds Docbank's current storage policy, restore writes it as a verified individual blob file instead. The result reports the loose count and grouped fallback reasons. This is a representation choice, not an integrity failure.
Individual files use the standalone daemon's compression policy, including when restoring through the embedded API: files of at least 4 KiB use zstd when it saves at least 10%. This includes large files reconstructed from backup chunks. The restored database and host extras remain ordinary files. Restoring an individual file temporarily needs space for its uncompressed bytes and its compressed candidate; the unused copy is removed when that file finishes. An overwrite restore reuses existing verified files in their current encoding.
Snapshots also carry a non-secret docbank-placement-v1 description of source
stores and the hashes held by each. Default restore places all verified content
in a fresh local primary store. It gives that store a new ID and ownership
epoch, the value used to distinguish one store owner from the next. It does not
inherit source paths, endpoints, credentials, buckets, bindings, or ownership
epochs.
--store-map maps source store IDs to binding profiles already loaded by the
daemon performing the restore. The TOML file must be an owner-private regular
file and may select a new empty namespace or a takeover. Mapped bytes are read
back independently before target authority is recorded. Unmapped bytes remain
local. A remote_only restore revokes primary catalog authority in the staged
database but leaves its physical staging files for garbage collection after
that database is published, so a failed overwrite cannot damage the existing
vault. Audited bytes also retain primary authority unless the mapping selects
both remote_only and allow_audited_remote_only. See
Multi-store storage for the file format and
trust boundary.
The mapping path is resolved on the daemon host, not on the caller's host. Mapping-file read and parse errors never include the file contents.
The restore API and explicit repo paths accept only loopback clients, because
the daemon opens and writes those server-local paths. A network client of a
daemon bound to a non-loopback address can create, list, and verify snapshots
in the configured [backup] repo; run restores on the daemon host, for example
with docker exec.
Backups exclude config.toml. Reconfigure processing profiles and other local
settings on the restored target before using them. Preserve the target's
storage bindings described below.
Restore also verifies that mapped filesystem stores do not overlap the live
source vault or backup repository. Before publication it gives an otherwise
empty target a minimal owner-private config.toml containing the mapped
binding profiles, so the restored vault can prove and read remote-only
authority after restart. An existing target configuration is preserved and
must already define the same mappings exactly.
What does the restore report mean?¶
Interactive restore shows metadata, document, extras, SQLite integrity, and
manifest-statistics progress as separate stages. --json suppresses progress
and returns one report containing physical layout counts and the
content_verified, sqlite_integrity, and manifest_stats proof fields. The
final report also inspects the restored vault's physical inventory while its
target-tree coordination is still held. It reports loose files, live packed
blobs, pack count, and any logically dead packed bytes that still occupy
immutable pack files. Those bytes have not been reclaimed. Inspect and repack
the restored target, not the currently running source vault:
DOCBANK_HOME=~/Restores/docbank-test docbank storage status
DOCBANK_HOME=~/Restores/docbank-test docbank storage repack
Reading this inventory does not run maintenance. If restore succeeds but the
inventory read fails, the report still records a successful restore and adds
a storage_warning.
A successful report means the target is a complete vault. Restore does not start it or replace the running vault with it. Inspect it under its own home first:
DOCBANK_HOME=~/Restores/docbank-test docbank vault verify
DOCBANK_HOME=~/Restores/docbank-test docbank document tree /
DOCBANK_HOME=~/Restores/docbank-test docbank daemon stop
Backups move forward only¶
Restore with the release that wrote the backup or a newer one. Older releases reject snapshots containing record kinds they do not understand instead of restoring a vault without those records.
Repository placement¶
Keep the repository outside $DOCBANK_HOME. It is independent archive state,
not a live-vault subdirectory. Its files are write-once, so a completed
repository is suitable for rsync, rclone, cloud-drive sync, filesystem
snapshots, or removable media. Sync after backup create completes. Never edit
repository packs, indexes, manifests, locks, or configuration by hand.