Skip to content

Importing Documents

docbank add is the bulk-migration path for decades of accumulated Documents/, Dropbox, and old-drive trees. It is designed to be run repeatedly against the same sources without ever duplicating a document or touching the originals.

What an import does

For each regular file:

  1. The content is hashed (SHA-256, streaming) and written to the blob store — durably, before any database row references it. Content that already exists in the vault is not written twice.
  2. One database transaction creates the tree node, its stable revision-one content_create version, the blob authority, and provenance: the original filesystem path and modification time survive any later renames or moves.

Directory arguments walk recursively. The directory's basename becomes a folder under --dest, and everything below keeps its relative structure:

docbank add ~/old-laptop/Documents --dest /archive
# → /archive/Documents/... mirrors the source tree

Trailing slashes and ./-style paths are normalized; add docs/ and add ./docs behave identically to add docs.

An explicitly named source may be a symlink to a directory. This supports ordinary platform layouts such as ~/Dropbox on macOS: docbank resolves that one root link, retains Dropbox as the virtual directory name, and records provenance using the path the user supplied. Symlinks encountered inside the tree remain skipped and reported, and an explicitly named symlink to a file is not imported.

Preflight a large tree

Inventory a source before Docbank opens any file content or changes the vault:

docbank add ~/Dropbox --preflight \
  --exclude .git \
  --exclude .Trash \
  --exclude project/cache

The report separates files currently eligible for packing (through 64 MiB), larger files that will remain authoritative loose objects, and files above the current format-v1 ingest ceiling. It also reports logical bytes, directory count, skipped non-regular entries, filesystem errors, and the largest groups by lowercase filename extension. Use --json for a structured, bounded report.

Preflight is metadata-only. In particular, it does not open cloud-provider placeholder content merely to estimate the import. That avoids an inventory silently hydrating an entire cloud tree, but it also means successful preflight cannot promise that every file will remain readable when the later import opens it. Re-run preflight after changing exclusions, then pass the exact same --exclude flags to the real docbank add command.

Filesystem names and provenance paths must currently be valid UTF-8. On POSIX filesystems that permit other byte sequences, preflight and ingest report each such entry with an escaped, printable path; Docbank does not open or import it, continues with the rest of the tree, and never alters the source.

A bare exclusion such as .git prunes that entry name wherever it occurs. A relative path such as project/cache prunes that path and its descendants within each supplied source. Exclusions do not use glob syntax. A pruned directory counts as one excluded entry because Docbank deliberately does not walk it to count hidden descendants. A trailing slash or leading ./ keeps a single-component rule path-shaped: cache matches that name anywhere, whereas cache/ and ./cache match only the source root's cache entry. Commas are literal filename characters, not rule separators; pass --exclude repeatedly for multiple rules.

Follow a long import

Human-mode docbank add performs a metadata-only scan to establish file and byte totals, then reports content-read progress while it imports:

docbank add ~/Dropbox --dest /archive --progress plain

auto (the default) draws a progress bar on a terminal and emits durable periodic lines when stderr is redirected. plain always emits durable lines; bar forces the redrawable form. Progress belongs on stderr and the terminal summary belongs on stdout. Use --json to suppress progress and emit only the machine-readable terminal report.

The scan totals are an estimate rather than a filesystem lock: a source may change before Docbank opens it. Byte progress counts content actually read, while a file counts as done only after its individual blob and metadata operation returns. Interrupting the command cancels the daemon request. Files that already completed remain authoritative and make a rerun converge; an incomplete file never receives node authority.

Idempotency: safe to re-run

Interrupted a 200,000-file import? Run the same command again. For each source file, docbank walks the candidate names in the destination directory — report.pdf, report (2).pdf, report (3).pdf, … — and:

  • if any live candidate has the same content, the file is counted as skipped (already imported, even if a prior run imported it under a suffix);
  • if all existing candidates have different content, the next free suffix is used;
  • otherwise the first free candidate name is taken.

Re-runs therefore converge instead of duplicating. Identity is content, not filename: the same bytes under two source names import as two nodes sharing one stored blob but carrying distinct version UUIDs.

Collisions

Two different files arriving at the same virtual name don't conflict — the newcomer is suffixed (scan.pdf → scan (2).pdf). The provenance record preserves where each one actually came from.

Inspect where a document came from

Docbank keeps provenance as durable metadata rather than leaving it hidden in an import log. Query a live file by virtual path or any retained file by stable node ID:

docbank provenance /archive/Documents/report.pdf
docbank provenance id:42 --json

The bounded newest-first result identifies the ingest, its source kind and description, the original source path and modification time, and the immutable SHA-256 identity of each provenance fact. active means no newer fact supersedes it; corrections retain earlier facts instead of rewriting them. Because a source path can disclose machine-local names, provenance is available only through the same authenticated API as the document itself.

This is inspection, not source management. Reading provenance neither opens nor changes the original file, and provenance alone does not pin a document version against ordinary retention or deletion policy.

Failures don't abort the batch

Unreadable files, permission errors, and non-regular files (symlinks, sockets, devices) are recorded and reported at the end; the rest of the import continues. A directory that can't be created in the tree (for example, its virtual path collides with an existing file) skips that subtree and continues with the next.

added: 4211  skipped: 12  failed: 2
failed: /src/broken.pdf: opening /src/broken.pdf: permission denied
failed: /src/link.pdf: not a regular file or directory (symlinks are skipped)

The exit code is non-zero when any file failed, so scripted migrations can detect partial imports. A missing or unreadable top-level source is reported the same way, and the command continues with the remaining source arguments.

Sources are read-only

Import never deletes or modifies source files, including a followed root directory symlink. Delete originals yourself once docbank verify and your own spot-checks satisfy you — the same archive-first, delete-later posture msgvault takes with mailboxes.

Remote API imports

Authenticated integrations can send one digest-checked file at a time through POST /api/v1/uploads. The server requires the writer's SHA-256 and byte length, computes both independently while streaming, and creates no node or blob authority when either differs. See the HTTP API and Agent Integration Guide for the exact contract.

Continuously ingest a local inbox

For directories that receive files over time, configure a daemon-owned [[watch]] entry instead of repeatedly running docbank add:

[[watch]]
name = "agent-sessions"
source = "~/agent-sessions"
destination = "/archives/agents"
settle_time = "30s"
minimum_age = "168h"
scan_interval = "5s"
exclude = ["cache/", ".DS_Store"]

[storage]
pack_interval = "1h"
pack_max_bytes = 268435456

The daemon observes a file's filesystem identity, size, and modification time for a full settle window before reading it. minimum_age = "168h" additionally requires seven days since the source's last modification, which is useful when an append-heavy session may pause for minutes or hours without being closed. The minimum-age gate survives restart; the settle observation deliberately does not, so every file still proves a complete unchanged window in the new daemon. Set minimum_age = "0s" or omit it for ordinary inboxes that need only the settle window.

Docbank then verifies that the confined source path still names the same unchanged object. It never follows entries that are symlinks and never changes or deletes source data. A time window cannot prove that a producer formally closed a file, so use a conservative age or point the watch at a completed-file handoff directory when one is available.

The watch name and slash-separated relative source path form a stable, portable provenance identity. The first stable observation creates the file under destination; later byte changes append immutable content versions to that same stable node. This remains true if a person or agent moves or renames the Docbank node after ingestion. Docbank remembers the last content accepted from the source independently of the node's current version, so an unchanged source does not overwrite a later edit or revert after daemon restart. Removing the source leaves the archived node alone. A renamed source-relative path is a new identity, not an implicit move.

For an agent-session archive, use the source tree itself for the organization you want to retain. For example, ~/agent-sessions/codex/project-alpha/2026/07/session-01.jsonl becomes /archives/agents/codex/project-alpha/2026/07/session-01.jsonl with the configuration above. Docbank does not reinterpret a vendor's session format: the relative path and source facts remain exact, while each accepted byte change becomes an immutable version of that document.

Use docbank provenance <path-or-id> to inspect the retained watch identity, source-relative path, and immutable supersession history for an imported node. JSONL session content up to the normal extraction limit is indexed by the built-in plain-text worker, so ordinary docbank search can find archived session text without a vendor-specific parser. The optional [storage] schedule packs accumulated small files with a finite per-run budget. It does not delete source files, prune versions, run GC, or rewrite existing packs. Portable backup and restore preserve the mirrored hierarchy, source provenance, every retained version, and its verified bytes.

The destination is exact rather than collision-suffixed. If unrelated content already occupies the intended path, or the previously mapped node is in the trash, the watcher fails closed instead of guessing. docbank jobs reports the named watch:<name> job and any terminal error; correct the problem and restart the daemon. Successful additions, updates, and unchanged observations appear in the daemon log. See Configuration for the complete field contract.