Importing documents¶

Use docbank add to copy files or entire folders into the vault. Docbank leaves the originals unchanged. After an interruption, run the same command again. It skips files whose content is already imported under a destination name.

Mailbox archives¶

For mailbox exports, use docbank mailbox import or Import mailbox in the web app. Both keep the original archive and give each message occurrence its own document, even when content or a Message-ID repeats. Decoded attachment documents are published with the message and its receipt.

docbank mailbox import ./All-mail.mbox --dest /mail --id mail-export --preview
docbank mailbox import ./All-mail.mbox --dest /mail --id mail-export
docbank mailbox watch mail-export
docbank mailbox receipts mail-export --after 0 --limit 100

To retry an interrupted upload, use the same --id, source bytes, and settings. Verified chunks are checked locally and skipped. The import job uses --id by default. Add a different --job-id to import that source into another destination without uploading it again.

The default dialect is mboxrd. Use --dialect mboxo for that format. The preview samples up to three messages, reads at most 16 MiB of expanded message data, and does not guess the dialect.

The background job checks a Google Takeout ZIP completely before it publishes any message. It refuses unsafe paths, symlinks, bad CRCs, and excessive expanded data, and it does not extract files to a caller-selected directory. Empty mailbox entries are skipped. A resumed job reuses the verified entry hashes and continues after its last committed message. Compressed entries still require decompression through that position.

Jobs survive daemon restart. mailbox cancel <id> stops a job, and mailbox resume <id> resumes a canceled or failed job. After each 100,000 messages, run mailbox continue <id> to scan the next segment of the same source into the same collection. Reports separate imported, rejected, pending, and canceled occurrences. A partial report with an unscanned tail is not a completed import. For another receipts page, pass the last returned ordinal as --after.

Source labels stay in occurrence provenance. To assign existing Docbank tags, opt in with --label-tag 'Project=existing-tag-id'. Mappings are job settings and cannot be changed. All mapped tags must exist when a job starts or resumes. A queued or running job prevents deletion of its mapped tags. If you delete a mapped tag while a job is paused, that job cannot resume. Start a new job with updated mappings. It imports the source again.

Importing does not grant remote-processing consent. Attachment documents may remain pending processing until a suitable profile and consent are configured. Mailbox import does not provide PDF email export.

Docbank stores each container as ordered, verified 64 MiB chunks, up to 256 GiB. Each MBOX or ZIP entry is limited to 50 GiB. A ZIP is limited to 1,000 entries and 200 GiB expanded data. Each emitted EML message is limited to 128 MiB, and malformed or oversized messages are recorded as rejected. There are at most two active uploads per owner and eight globally. Incomplete uploads expire 24 hours after creation. Sealed sources have no expiry. Globally, at most two jobs run at once and at most eight are queued or running. Each owner can have at most two queued or running. These limits bound work. They are not throughput claims.

Explicit EML transfers¶

An exporter can register an archive and supply its own stable occurrence reference without connecting Docbank to the source application:

docbank mailbox register synthetic-export 'Explicit exported messages'
docbank mailbox transfer ./message.eml --archive synthetic-export --reference item-1

A retry with equal bytes returns the original receipt. Changed source bytes require --if-rev with the receipt's target revision, and the earlier content version is preserved. Editing or remapping the target causes a conflict. Trashing it returns a tombstone and does not resurrect it.

Source mappings, referenced versions, attachment relations, and sealed container bytes survive portable backup and restore and have no silent retention deadline. There is no release command for these retry guarantees, and referenced versions and relations cannot be silently pruned. Emptying trash skips these messages, attachments, and containing folders while deleting unrelated eligible trash. Ordinary docbank add message.eml does not create an external transfer identity.

Ordinary file imports¶

For each regular file, Docbank performs two steps:

1. Docbank computes the SHA-256 content hash while reading the file. It stores the bytes durably before adding database records. Identical content already stored in the vault is reused.
2. Docbank creates the file entry, its revision-one content_create version, the record of its stored content, and its provenance in one database transaction. Provenance records the original path and modification time. These facts survive later renames and moves.

See Storage for the content records and Editing and versions for version identity.

MIME type detection¶

Automatic MIME selection for local imports, docbank put, and load-file package staging inspects the first 512 bytes with Docbank's pinned signature detector. Recognized signatures take priority over the host's extension table, including JPEG, HEIC, and HEIF. docbank put --mime-type still overrides automatic selection. The .eml rule runs first and always uses message/rfc822.

An extension can refine a broad detector result when it names the same MIME node, an alias, a child format, or another member of the text/plain family. Unknown bytes, including empty files, can use a valid extension mapping as their type. An unrelated, invalid, or binary ancestor mapping leaves the recognized detector result in place. Compatible resolver parameters remain on the selected value.

The importer also has fixed suffix refinements for detector results that the pinned library cannot name as a subtype:

• TIFF bytes with .arw, .dng, .cr2, or .nef use image/x-sony-arw, image/x-adobe-dng, image/x-canon-cr2, or image/x-nikon-nef.
• Unknown .raf bytes use the fixed image/x-fuji-raf type.
• PNG or APNG bytes with .png or .apng use image/png and the existing preview path.
• Matroska bytes with .mka use audio/x-matroska.
• Text-family bytes with .xmp use application/rdf+xml, including XMP packets without an XML declaration.
• Text-family bytes with .md or .markdown use text/markdown.
• Text-family bytes with .go, .rst, .yaml/.yml, or .tex use text/x-go, text/x-rst, application/yaml, or application/x-tex.

The pinned detector sees these TIFF-based RAW formats as image/tiff. The suffix rules add their stored subtype without trusting an arbitrary host mapping. Automatic local replacements and package staging use the same selector. Each content version keeps the MIME observation selected when that version was created. Existing versions are not rewritten.

Directory arguments walk recursively. The directory's basename becomes a folder under --dest, and everything below keeps its relative structure:

docbank add ~/old-laptop/Documents --dest /archive
# → /archive/Documents/... mirrors the source tree

Trailing slashes and ./-style paths are normalized, so add docs/ and add ./docs behave identically to add docs.

An explicitly named source may be a symlink to a directory. This supports ordinary platform layouts such as ~/Dropbox on macOS. Docbank resolves that one root link, keeps Dropbox as the virtual directory name, and records provenance using the path the user supplied. Symlinks encountered inside the tree are still skipped and reported, and an explicitly named symlink to a file is not imported. Entries filtered out by an include or exclude rule are not failures. Selected non-regular entries are reported as failures.

Preflight a large tree¶

Inventory a source before Docbank opens any file content or changes the vault:

docbank add ~/Dropbox --preflight \
  --include '*.pdf' \
  --exclude .git \
  --exclude .Trash \
  --exclude project/cache

The report separates files currently eligible for packing (through 64 MiB), larger files that will remain individual stored files, and files above the current format-v1 ingest ceiling. It also reports logical bytes, directory count, skipped non-regular entries, filesystem errors, and the largest groups by lowercase filename extension. Use --json for a structured report.

Preflight reads metadata only. It does not open cloud placeholders: file entries whose contents still need to be downloaded from a provider. This lets you estimate an import without downloading the whole tree. A successful scan does not guarantee that the later import can read every file.

On macOS, cloud placeholders counts regular files whose bytes are not local, including iCloud Drive and Google Drive for Desktop placeholders. These files also count toward the report's size classes. Check this count before starting an import that may require substantial downloading.

A provider may decline to download a placeholder's contents for the process that opens it. The usual case is a daemon started by launchd as a background job, while an interactive session succeeds. Docbank reports that failure for the individual file, names the cause, and suggests opening the file once from a user session (or marking it available offline) before retrying the import. Re-run preflight after changing selection, then pass the same --include and --exclude flags to the real docbank add command.

Filesystem names and provenance paths must currently be valid UTF-8. On POSIX filesystems that permit other byte sequences, preflight and ingest report each such entry with an escaped, printable path. Docbank does not open or import it, continues with the rest of the tree, and never alters the source.

Choose files with include and exclude rules¶

Use include rules to select files and exclude rules to skip files or whole subtrees. Exclusions win. Include rules leave directories open for traversal.

Rules use Go's path.Match grammar. * and ? do not cross /, and ** does not mean recursive matching. Use / separators on every platform. Backslashes are rejected. Use bracket expressions to match a literal [, ?, or *. Matching is case-sensitive, including on Windows.

Repeat a flag for each rule. Commas are literal characters. Empty rules, absolute paths, parent traversal, and malformed patterns are rejected before the walk. Watched-inbox exclusions remain literal and do not use this glob syntax.

When the source argument is a single file, a basename rule such as *.pdf matches it. A path-form rule such as reports/*.pdf applies to a directory source's relative paths.

Label and browse one import run¶

The HTTP ingest body can attach an optional label to the logical run. The label publishes atomically with the first committed document observation:

curl -sS -X POST -H "X-Api-Key: $DOCBANK_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"paths":["/srv/import/review"],"dest":"/archive","collection_label":"Review set"}' \
  http://127.0.0.1:43210/api/v1/ingest

For streamed progress, send the same field to the streaming route:

curl -sS -N -X POST -H "X-Api-Key: $DOCBANK_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"paths":["/srv/import/receipts"],"dest":"/archive","collection_label":"Receipt batch"}' \
  http://127.0.0.1:43210/api/v1/ingest/stream

The terminal report includes ingest_id when at least one file committed. Use that ID with GET /api/v1/collections/{id} and GET /api/v1/collections/{id}/members. List current nonempty runs with GET /api/v1/collections. Collection membership follows the documents as they move within the vault and reports current paths, sizes, and versions. Trashed files and superseded provenance disappear from live membership. Caller-supplied embedded: provenance never creates a collection.

The label has a separate ETag and edit route. Read GET /api/v1/collections/{id}/label, then PUT exactly {"label":"New name"} or {"label":null} to the same path with its quoted revision in If-Match. Non-null labels stay unique even while a collection is empty. After permanent audit is enabled, label changes fail with HTTP 409 and audit_mutation_unsupported. The existing label and collection remain readable.

If only some files succeed, the receipt names the real run and lists failures beside it. If nothing commits, the receipt omits ingest_id and no collection or label is created. Label collisions and audit restrictions on an initial label use this same per-file failure list instead of turning the whole batch into one transport error.

Follow a long import¶

Human-mode docbank add performs a metadata-only scan to establish file and byte totals, then reports content-read progress while it imports:

docbank add ~/Dropbox --dest /archive --progress plain

auto (the default) draws a progress bar on a terminal and prints periodic progress lines when stderr is redirected. plain always prints lines, and bar forces the redrawn bar. Progress goes to stderr and the final summary goes to stdout. Use --json to suppress progress and emit only the machine-readable final report.

The scan totals are an estimate, because a source may change before Docbank opens it. Byte progress counts content actually read. A file counts as done only after its blob and metadata operation returns. Interrupting the command cancels the daemon request. Docbank keeps files that completed successfully and skips them on a rerun. It does not create a file entry for an incomplete import.

What happens when I run the import again?¶

If a 200,000-file import is interrupted, run the same command again. For each source file, Docbank walks the candidate names in the destination directory (report.pdf, report (2).pdf, report (3).pdf, …) and:

• if any live candidate has the same content, the file is counted as skipped (already imported, even if a prior run imported it under a suffix);
• if all existing candidates have different content, the next free suffix is used;
• otherwise the first free candidate name is taken.

Repeating the same source import does not create extra copies of its entries. Two differently named source files with identical bytes still import as two file entries. They share stored content but have distinct version UUIDs.

Each explicit filesystem re-run is still a new logical ingest run. When bytes already match a destination node, Docbank adds that existing node to the new run without creating a content version. Recording this new membership advances the node revision, even without a collection label or any flags. This also applies to --replace when the bytes are identical. API clients holding the old ETag must refresh it before their next write. Otherwise If-Match returns 412 stale_revision. Repeating the same observation within one run is a no-op.

Digest-checked POST /uploads retries are different. An equal retry returns the existing node with an unchanged revision and does not return an unused ingest identity.

Collisions¶

Two different files arriving at the same virtual name don't conflict. The newcomer gets a suffix (scan.pdf → scan (2).pdf). The provenance record preserves where each one came from.

Replace a changing local file¶

Use --replace when a repeated local add represents one changing source:

docbank add ~/reports/summary.pdf --dest /archive --replace

Docbank resolves the exact destination name and records its node revision before reading the source. Different bytes become a new content version on the same node. Unchanged bytes count as skipped and keep the stored MIME type and version history, while the new run membership advances the node revision as described above. A live directory fails that file before source content is opened. If an absent destination is claimed while the source is read, the create reports a conflict and does not choose a suffix. A stale observed revision reports a conflict and leaves the newer content current. Omit --replace for ordinary collision suffixing.

Inspect where a document came from¶

Docbank keeps the facts about where a document came from as provenance. Query a live file by vault path, or any retained file by stable node ID:

docbank provenance /archive/Documents/report.pdf
docbank provenance id:42 --json

The newest-first result identifies the ingest, its source kind and description, the original source path and modification time, and the SHA-256 identity of each provenance fact. active means no newer fact supersedes it. A correction adds a fact and keeps the earlier ones. Because a source path can disclose machine-local names, provenance is available only through the same authenticated API as the document itself.

Reading provenance does not open or change the original file. A provenance record also does not prevent ordinary retention or deletion rules from removing a document version.

Applications can append an origin learned later through the HTTP API or embedded Go API. They can correct an active caller-supplied fact by adding a new fact that supersedes it. CLI and watched ingest facts cannot be superseded because Docbank uses them to recognize repeated imports. Append an additional origin instead. The provenance CLI command reads this history.

Failures don't abort the batch¶

Unreadable files, permission errors, and non-regular files (symlinks, sockets, devices) are recorded and reported at the end, and the rest of the import continues. A directory that can't be created in the tree (for example, its virtual path collides with an existing file) skips that subtree and continues with the next.

added: 4211  skipped: 12  failed: 2
failed: /src/broken.pdf: opening /src/broken.pdf: permission denied
failed: /src/link.pdf: not a regular file or directory (symlinks are skipped)

The exit code is non-zero when any file failed, so scripted migrations can detect partial imports. A missing or unreadable top-level source is reported the same way, and the command continues with the remaining source arguments.

Sources are read-only¶

Import never deletes or modifies source files, including a followed root directory symlink. Before deleting originals yourself, run docbank verify, spot-check the imported documents, and capture a backup.

Remote API imports¶

Authenticated integrations can send one digest-checked file at a time through POST /api/v1/uploads. The server requires the writer's SHA-256 and byte length, computes both independently while streaming, and creates no node or blob record when either differs. See the HTTP API and Agent integration guide for the contract.

Backups include collection labels and their revisions, run membership, and the audit history for repeated operational observations. Older readers that do not understand these records reject such a snapshot instead of restoring it without the collection records.

Continuously ingest a local inbox¶

For directories that receive files over time, configure a daemon-owned [[watch]] entry instead of repeatedly running docbank add:

[[watch]]
name = "agent-sessions"
source = "~/agent-sessions"
destination = "/archives/agents"
settle_time = "30s"
minimum_age = "168h"
scan_interval = "5s"
exclude = ["cache/", ".DS_Store"]

[storage]
pack_interval = "1h"
pack_max_bytes = 268435456

The daemon observes a file's filesystem identity, size, and modification time for a full settle window before reading it. minimum_age = "168h" additionally requires seven days since the source's last modification, which is useful when an append-heavy session may pause for minutes or hours without being closed. The minimum-age gate survives restart. The settle observation does not, so after a restart every file must again stay unchanged for a full settle window. Set minimum_age = "0s" or omit it for ordinary inboxes that need only the settle window.

Docbank then verifies that the confined source path still names the same unchanged object. It never follows entries that are symlinks and never changes or deletes source data. A time window cannot prove that a producer formally closed a file, so use a conservative age or point the watch at a completed-file handoff directory when one is available.

The watch name and slash-separated relative source path form a stable, portable provenance identity. The first stable observation creates the file under destination. Later byte changes append content versions to that same node. This remains true if a person or agent moves or renames the Docbank node after ingestion. Docbank remembers the last content accepted from the source independently of the node's current version, so an unchanged source does not overwrite a later edit or revert after daemon restart. Removing the source leaves the archived node alone. Renaming a source-relative path creates a new identity. Docbank does not treat it as a move.

For an agent-session archive, use the source tree itself for the organization you want to retain. For example, ~/agent-sessions/codex/project-alpha/2026/07/session-01.jsonl becomes /archives/agents/codex/project-alpha/2026/07/session-01.jsonl with the configuration above. Docbank does not interpret a vendor's session format. It keeps the relative path and source facts unchanged, and each accepted byte change becomes a new version of that document.

Use docbank provenance <path-or-id> to inspect the watch identity, source-relative path, and supersession history of an imported node. JSONL session content up to the normal extraction limit is indexed by the built-in plain-text worker, so ordinary docbank search can find archived session text without a vendor-specific parser. The optional [storage] schedule packs accumulated small files with a finite per-run budget. It does not delete source files, prune versions, run GC, or rewrite existing packs. Portable backup and restore preserve the mirrored hierarchy, source provenance, every retained version, and its verified bytes.

The watcher uses the exact destination name and does not add a collision suffix. It stops with an error if unrelated content already occupies that path or the previously mapped node is in trash. docbank jobs reports the named watch:<name> job and any terminal error. Correct the problem and restart the daemon. Successful additions, updates, and unchanged observations appear in the daemon log. See Configuration for the complete field contract.