Embed in Go¶

Use go.kenn.io/docbank to own a document vault inside your Go process. Your application can store, version, retrieve, and back up documents without starting or discovering a daemon. The embedded API uses the same tree, content checks, storage rules, and exclusive vault lock as standalone Docbank.

Use an embedded vault when the application itself should own document lifecycle. Use the HTTP API when independent processes need to share one standalone vault.

Inspect format coverage¶

Vault.FormatCoverage returns the immutable capability snapshot captured after the configured rendition providers were constructed. Vault.LookupFormat resolves a catalog ID or extension against that same snapshot:

coverage, err := vault.FormatCoverage(ctx)
if err != nil {
    return err
}
lookup, err := vault.LookupFormat(ctx, "wpd")
if err != nil {
    return err
}

Returned maps and slices are defensive copies. Changing them does not alter later calls. Both methods follow the normal vault lifecycle and return ErrClosed after Vault.Close. See Format coverage for the seven capabilities and five states.

Transfer explicitly exported email¶

Call Vault.RegisterMailboxArchive(ctx, id, description) once for a stable source archive identity. Then call Vault.TransferEML(ctx, request, reader) with a docbank.MailboxTransferRequest: archive ID, stable occurrence reference, SHA-256, byte size, settings identity, destination directory ID and document name. Docbank reads only the supplied reader. It does not discover source applications, accounts or host paths.

The returned receipt binds the ordinary message, decoded attachment documents and exact retained versions. Equal retries return that receipt. Changing the source requires ExpectedRevision. A changed target conflicts, and a trashed target returns a tombstone. These calls use the vault's existing lifecycle and mutation ownership. See Explicit EML transfers.

Create a vault service¶

Each root is an independent archive. One process may open several non-overlapping roots at once. The same root, an ancestor, or a descendant is rejected while another daemon, embedded vault, or restore target owns it, even when a filesystem alias spells that tree differently.

package archive

import (
    "bytes"
    "context"
    "io"

    "go.kenn.io/docbank"
)

func StoreSession(ctx context.Context, root string, sessionID string, jsonl []byte) error {
    vault, err := docbank.New(ctx, docbank.Config{Root: root})
    if err != nil {
        return err
    }
    defer vault.Close()

    _, err = vault.Put(
        ctx,
        "/sessions/"+sessionID+".jsonl",
        bytes.NewReader(jsonl),
        docbank.PutOptions{MediaType: "application/x-ndjson"},
    )
    if err != nil {
        return err
    }

    content, err := vault.OpenContent(ctx, "/sessions/"+sessionID+".jsonl")
    if err != nil {
        return err
    }
    defer content.Reader.Close()
    _, err = io.Copy(io.Discard, content.Reader)
    if err != nil {
        return err
    }
    return content.Reader.Verify()
}

An embedded application can append an origin learned after ingestion with the same immutable fact model:

appended, err := vault.AppendProvenance(ctx, receipt.Node.ID,
    docbank.ProvenanceAppendOptions{
        IfRevision: receipt.Node.Revision,
        Source: docbank.ProvenanceSource{
            Kind: "agent", Description: "triage", Reference: "opaque://laptop/report.txt",
        },
    })

The result contains the updated node, live path, appended fact, and receipt. Set a positive IfRevision to require the node revision you inspected. Zero makes the operation unconditional.

To correct an active caller-supplied fact on the same node, set Supersedes to that fact's identity. Docbank records the source reference exactly as supplied and never opens it.

Put creates missing virtual directories. Repeating the same bytes and media type resolves to the current version. Changed bytes append an immutable content version and keep the node ID. Supply PutOptions.Expected when the caller already knows the SHA-256 and byte count and wants Docbank to reject a mismatched stream before it commits any metadata. Supply a positive PutOptions.IfRevision when replacing content derived from an earlier read. Docbank rejects the write with ErrStaleRevision if the file has changed since that revision, including when the new bytes happen to match the current head.

Use Create when an application owns an immutable key and must never replace different content already stored there. It requires the expected SHA-256 and size. An exact retry is idempotent. A different byte identity, media type, or node kind returns ErrContentConflict without appending a version:

receipt, err := vault.Create(ctx, "/records/immutable.jsonl", reader,
    docbank.CreateOptions{
        MediaType: "application/x-ndjson",
        Expected: docbank.ContentIdentity{SHA256: digest, Size: size},
        Provenance: &docbank.ProvenanceSource{
            Kind:        "agent-session",
            Description: "local session archive",
            Reference:   "sessions/01K123.jsonl",
        },
    },
)

CreateOptions.Provenance lets an embedded owner record where an immutable document came from without inventing application-specific Docbank tables. The kind and description identify the source system. The reference is an opaque source-local URI, archive key, path, or other stable identifier. An optional ModifiedAt records the source's own timestamp in canonical UTC form. Docbank does not interpret the reference or grant it retention authority.

For a newly created document, the node, first content version, ingest record, and provenance fact commit as one metadata transaction. If any part fails, none is committed. An exact Create retry succeeds only when its active provenance also matches, so a caller cannot accidentally claim source evidence that was never recorded. Superseding that matching fact makes the original request return ErrContentConflict unless another active fact still matches. A request matching the corrected provenance remains idempotent when the content and media type are unchanged.

AppendProvenance may supersede active caller-supplied facts on the same node, including facts supplied through CreateOptions.Provenance. Operational CLI and watched-folder ingest facts cannot be superseded, because they keep re-ingest idempotent. Append newly learned origins alongside them instead. Use Provenance to page through the history, newest first:

page, err := vault.Provenance(ctx, receipt.Node.ID, docbank.ProvenanceOptions{
    Limit: 100,
})
if err != nil {
    return err
}
for _, fact := range page.Items {
    // fact.SourceReference remains the exact opaque identifier supplied above.
    _ = fact
}

The node, live path, total, and page come from one read snapshot. Path is empty for a trashed node. Provenance records origin. It does not prevent trash empty, version pruning, or garbage collection. Applications must manage any retention policy for external references themselves.

ContentIdentity always describes decoded document bytes, regardless of whether those bytes are stored raw, zstd-compressed, or in a pack. SHA-256 is the canonical logical identity, not a digest of a physical storage file.

Configure document processing¶

Pass named ProcessingProfileConfig values through Config.Processing.Profiles. Each value binds a canonical portable profile to its provider implementations. EmbeddingProviders, EmbeddingDisclosures, and Tokenizers use the embedding binding names from that profile.

Every provider with an operator_network or hosted_provider trust boundary requires a ProcessingRuntimeDisclosure.Endpoint when docbank.New opens the vault. Set RenditionDisclosure for a rendition provider and an entry in EmbeddingDisclosures for each network embedding binding. Opening fails if an endpoint is missing or invalid. Docbank cannot infer it from a provider interface.

The endpoint must describe the provider's configured HTTP(S) destination and contain no URL credentials, query parameters, or fragment. A local_process provider accepts an empty endpoint or in-process. Docbank fills omitted processor and deployment identities from the provider and profile, and derives metadata classes and retained artifact roles. Review the resulting PlanProcessing response before granting consent. These disclosures are part of its fingerprint.

For example, given a canonical profile with a rendition provider and a text embedding binding, the matching constructed providers, and that binding's tokenizer:

vault, err := docbank.New(ctx, docbank.Config{
    Root: root,
    Processing: docbank.ProcessingOptions{
        Profiles: map[string]docbank.ProcessingProfileConfig{
            "private": {
                Profile: profile,
                RenditionProvider: renditionProvider,
                RenditionDisclosure: docbank.ProcessingRuntimeDisclosure{
                    Endpoint: "http://127.0.0.1:5001",
                },
                EmbeddingProviders: map[string]document.EmbeddingProvider{
                    "text": embeddingProvider,
                },
                EmbeddingDisclosures: map[string]docbank.ProcessingRuntimeDisclosure{
                    "text": {Endpoint: "http://127.0.0.1:8080"},
                },
                Tokenizers: map[string]document.Tokenizer{"text": tokenizer},
            },
        },
    },
})
if err != nil {
    return err
}
defer vault.Close()

Use the same endpoints when constructing the providers. A disclosure names a destination but does not configure its transport. Import document from go.kenn.io/docbank/document. See Document understanding in Go for provider construction and Document processing for planning, consent, retained results, and the private-deployment acceptance runner.

Set DocumentSearchRequest.ContentFirst to prefer content matches, following the HTTP search ordering contract.

Set DocumentSearchRequest.MediaSources to search only the transcripts of exact recording sources. Reports then set MediaSourceSelection and list MediaSelections, and evidence names its MediaSources, following the HTTP media source contract. Invalid selectors, or a mode or reranking choice they do not support, match docbank.ErrMediaSearchInvalid.

Read exact evidence windows¶

Use ReadEvidenceWindow to read a bounded excerpt from a rendition you already identified. Keep the file's vault, node, content-version and content SHA-256 alongside the attachment, build and sanitized-Markdown SHA-256 returned by Rendition or document detail metadata. Pass all seven identities in an EvidenceWindowRequest:

window, err := vault.ReadEvidenceWindow(ctx, docbank.EvidenceWindowRequest{
    VaultUID: vault.ID(), NodeID: receipt.Node.ID,
    ContentVersionID: receipt.Version.ID, ContentSHA256: receipt.Computed.SHA256,
    RenditionAttachmentID: rendition.AttachmentID,
    BuildID: rendition.BuildID, RenditionSHA256: rendition.SHA256,
    Offset: 0, MaxChars: 8_000,
})
if err != nil {
    return err
}
// Keep window.NextOffset to continue from the exclusive end of this window.

Offsets count Unicode scalars. Zero MaxChars selects 8,000. Other values must be 1 through 16,000. ActualStart, ActualEnd, and NextOffset describe the returned range, and ResponseBytes counts its UTF-8 bytes. A read starting at EOF returns empty text with EOF: true. An offset beyond EOF matches ErrInvalidRenditionWindow. Malformed references match ErrInvalidEvidenceRequest.

Only the current, live content version and its active rendition are readable. A rename preserves the reference. A content or rendition replacement, trash, pruning, or retirement makes the old reference match ErrEvidenceUnavailable. The read never runs processing, selects a newer rendition, or pins retention. The vault lifecycle lease covers the internal blob read and cleanup and is released before the function returns. Callers receive text, not an open stream.

The returned digest is the one the catalog records. A window read can end before whole-artifact checksum verification reaches EOF. Search chunk and embedding segment offsets use their own coordinates. Do not use them as Markdown offsets. Start a document overview at zero, or keep offsets from an earlier window. See the HTTP evidence contract for the equivalent authenticated route.

Retain and process a remote recording¶

Embedded callers can keep the recording identity separate from the protected reference used to obtain a local file. Submission stores the pending occurrence and performs no network access. The canonical_url field is optional for configured-origin requests, and is the identity input for a generic URL reference when present. See the canonical URL rules for the permanent source identity. A Cap Cloud share or embed URL uses one Cap source per video ID. A Loom share or embed URL uses a route-qualified source because Loom does not document equality between those routes. Submission keeps the Loom reference unsupported and performs no network access.

remote, err := vault.SubmitRemoteRecording(ctx, docbank.RemoteRecordingRequest{
    OperationID:  "00000000-0000-4000-8000-000000000451",
    ReferenceURL: "https://private.invalid/share/call?token=synthetic",
    CanonicalURL: "https://recordings.invalid/share/call",
    Occurrence: docbank.MediaOccurrenceInput{
        Ref: "call-1", Revision: "1", Filename: "call.wav",
    },
})
if err != nil {
    return err
}

If that reply is lost, Vault.RemoteRecordingReceipt reads the saved receipt by operation ID. ErrNotFound describes only that read. A submission still in flight can commit afterward.

Obtain the file through the caller's own approved path, then attach it with its SHA-256 and byte count. WAV and MP3 files use the existing media rules. Remote MP4 originals use video/mp4, a .mp4 filename, and limits of 20 MiB, 2,088,960 coded pixels, 300,000 milliseconds, and 18,000 frames. The service's default byte limit is 512 MiB, its hard maximum is 1 GiB, and the audio duration limit is 24 hours.

original, err := vault.ImportRecordingArtifact(
    ctx, docbank.MediaArtifactRequest{
    OperationID: "00000000-0000-4000-8000-000000000452",
    SourceID:    remote.SourceID,
    OccurrenceID: remote.OccurrenceID,
    Kind:        "media",
    Origin:      "supplied",
    Filename:    "call.wav",
    MediaType:   "audio/wav",
    SHA256:      recordingHash,
    ByteLength:  recordingSize,
    Content:     recording,
})
if err != nil {
    return err
}

Import a caption or transcript only after the original is bound. A caption uses application/x-subrip and the built-in supplied-captions profile. It keeps cue timing and supplied provenance and supports lexical and auto search. Semantic and hybrid search are not configured for supplied captions. A transcript uses the built-in supplied-transcript profile after a plan and consent grant.

caption, err := vault.ImportRecordingArtifact(
    ctx, docbank.MediaArtifactRequest{
    OperationID: "00000000-0000-4000-8000-000000000453",
    SourceID: remote.SourceID, OccurrenceID: original.OccurrenceID,
    Kind: "caption", Origin: "supplied", Provider: "loom", Filename: "captions.srt",
    MediaType: "application/x-subrip", SHA256: captionHash,
    ByteLength: captionSize, Content: captionReader,
})
if err != nil {
    return err
}
selector := docbank.ProcessingSelector{
    NodeID: nodeID, ContentVersionID: original.ContentVersionID,
    Profile: "supplied-captions",
}
plan, err := vault.PlanProcessing(ctx, docbank.ProcessingPlanRequest{
    Selector: selector,
})
if err != nil {
    return err
}
if _, err := vault.GrantProcessingPlanConsent(
    ctx, docbank.ProcessingConsentGrantRequest{
    PlanRequest: docbank.ProcessingPlanRequest{Selector: selector},
    PlanFingerprint: plan.Fingerprint,
}); err != nil {
    return err
}
receipt, err := vault.RetryMedia(ctx, "00000000-0000-4000-8000-000000000454",
    remote.SourceID, docbank.MediaProcessingRequest{
        Profile: "supplied-captions",
        SuppliedInputID: caption.SuppliedInputID,
    })

RetryMedia selects the caller's newest visible occurrence for the source and returns once the work is durably queued. It cannot select an older occurrence. Use PlanProcessing and StartProcessing with that recording's node and current content version instead. Read MediaStatus for the newest attempt and its coverage. Coverage follows the source version selected by the visible occurrence, so a transcript for older bytes cannot cover a later recording revision. A recognized Cap Cloud or Loom link returns unsupported even when Acquire is set, because neither manual reference path has a supported automatic download owner. Import the caller-held file with ImportRecordingArtifact. Other remote acquisition remains unavailable. Origin registration is daemon configuration. Embedded vaults register no self-hosted Cap origins, so the embedded API does not apply their registered origin identity rules.

Read the published transcript with the same ID tuple used for processing:

transcript, err := vault.MediaTranscript(ctx, docbank.MediaTranscriptRequest{
    SourceID: remote.SourceID,
    SourceVersionID: receipt.SourceVersionID,
    ContentVersionID: receipt.ContentVersionID,
})

EvidenceState is independent of CoverageState and OperationState. EvidenceState == "ready" is the only state with text. Units preserve timing and speaker facts, and Origin distinguishes supplied from generated evidence. A mismatched content version returns stale without text, so callers must check the complete tuple before displaying a transcript.

Extract and read source metadata¶

EnsureSourceMetadata verifies and processes one immutable content version with Docbank's current local extractor, then returns its typed metadata. The result carries the complete ContentVersion, so callers can bind the facts to the SHA-256 and node version they describe:

metadata, err := vault.EnsureSourceMetadata(ctx, receipt.Version.ID)
if err != nil {
    return err
}
for _, field := range metadata.Fields {
    // Interpret fields by namespace, key, and typed value.
    _ = field
}

EnsureSourceMetadata runs locally and returns when extraction finishes. It processes only the requested version and reuses the current extractor's result when one exists. It does not start background workers.

The method holds the vault's mutation lock during extraction. It verifies the complete original, including large media files, then applies the format-specific parsing limits. Concurrent Put, Create, and maintenance calls wait. If Docbank cannot open or verify the source bytes, it returns ErrContentUnavailable.

Use SourceMetadata for a read-only lookup. It returns ErrNotFound when no result has been published. Both methods return all local fields, including sensitive fields. Your application decides which fields it may disclose.

Publish email attachment documents¶

Use EnsureEmailMetadata to retain the MIME inventory for one email version. Then call PublishEmailDocuments with a document.EmailDocumentPublicationRequest: an operation ID, the parent node/version/hash/size, the returned generation and attachment IDs, and a destination directory ID with its current revision. No host filesystem path is accepted. This operation does not run automatically on import.

Publication verifies retained payload bytes and creates ordinary file documents for the full known attachment inventory in one transaction. Inline resources are included, but body alternatives and multipart containers are excluded. An attached email becomes a child email, whose own attachments can be published separately. An explicitly attached single-part root also becomes a child. Parts inside encrypted containers stay encrypted relations without child files. The receipt says whether the MIME inventory is complete or partial. It records unavailable, unsupported, failed, or encrypted parts and does not create empty files for them.

Equal bytes share storage but get separate document occurrences. Filenames use the safe MIME name plus operation and occurrence identifiers. A reuse selection may instead identify an existing child version with matching bytes and its node revision. Raw filenames remain in the MIME inventory. Renames, later content versions, and decoder reprocessing never move an old relation to a different version.

Keep the operation ID and request unchanged when retrying. The same request returns the original receipt, including after backup and restore. A changed request conflicts. EmailDocumentPublication reads that receipt. EmailDocumentRelations selects either one parent version or one child version and returns current processing status with each relation. Pages default to 100 rows, accept at most 250, include a total, and return the next operation/order pair when another page exists. Publication accepts at most 1,000 MIME parts, 128 MiB per payload, and 256 MiB of decoded payloads. HTTP request and response bodies are limited to 2 MiB.

Publication itself does not prove indexing or authorize provider disclosure. RequestEmailDocumentProcessing submits one receipt occurrence to the ordinary rendition scheduler with a processing profile, execution identity, artifact policy, principal, scope, and consent classes. Existing consent must authorize those exact inputs and retained outputs. GrantProcessingConsent grants that consent. RevokeProcessingConsent advances the principal/scope revocation fence. Publication and processing requests never grant consent or inherit it from a parent email. The profile must match a configured provider. Docbank inspects the child bytes and rejects execution metadata that differs from the prepared upload before enqueueing work. The embedded method runs the provider and waits for completion. The HTTP endpoint returns after enqueueing for the daemon worker. Both paths check consent again before provider access and publication.

Relation status reports indexed for complete searchable output from ordinary text extraction or an active rendition of that child version. Partial or truncated renditions report partial, and empty output reports none. Text-extraction failures report failed with reason text_extraction_failed. Completed jobs without serving output report decoded with reason rendition_not_serving. Other states include pending, unsupported, encrypted, and unavailable. Ordinary search returns child matches. QueryV1 does not add an email-family traversal predicate.

Trash and restore keep relations and do not cascade to children. A referenced version cannot be permanently deleted or pruned. A targeted purge of a referenced MIME inventory conflicts. A vault-wide purge skips it and purges other eligible derivatives. Conflicts identify the blocking publication operation. To release those references, call RemoveEmailDocumentPublication with the operation ID and its request digest. This removes that receipt and its relations, preserves ordinary children, and gives up the operation's retry guarantee. Other receipts keep their own references. Ordinary audit and content-retention rules still apply. Use a fresh operation ID for a new publication after release, or reuse the existing child versions. A generated filename collision returns email_document_conflict and leaves the publication uncommitted.

CLI users can inspect and release blocking receipts through the daemon with docbank email-documents.

The typed client names the principal/scope consent operations GrantScopedProcessingConsent and RevokeScopedProcessingConsent to distinguish them from consent for a configured processing plan.

The authenticated HTTP and typed client surfaces expose the same operations:

These routes require the master API key and are outside the current browser session capability. All relation and receipt DTOs live in go.kenn.io/docbank/document. Consumers can use the same identities for navigation and downloads without inferring parentage from names or hashes.

Read canonical visual previews¶

EnsureVisualPreview synchronously processes one immutable content version when the current built-in recipe has no recorded result. It returns the active preview. If the recipe was already recorded, another recipe's active result stays selected. Ready results identify the preview bytes and dimensions. Unsupported and failed results carry a stable failure code and no content.

preview, err := vault.EnsureVisualPreview(ctx, versionID)
if err != nil {
    return err
}
if preview.State == document.VisualPreviewReady {
    content, err := vault.OpenVisualPreview(ctx, versionID)
    if err != nil {
        return err
    }
    defer content.Reader.Close()
    // Reach EOF or call Verify before trusting the bytes.
}

OpenVisualPreview uses the same verified-reader contract as original content. It returns ErrVisualPreviewUnavailable for a cataloged unsupported or failed result and ErrNotFound when no preview result exists. VisualPreview remains a read-only lookup. Opening an embedded vault does not start a preview worker. Applications choose when to call the synchronous producer. The built-in producer supports JPEG, PNG, GIF, still WebP, and supported embedded JPEG previews in ARW, DNG, CR2, NEF, and RAF camera RAW files. See Visual previews for format limits, output size, and recipe selection.

Use VisualPreviewForSize, EnsureVisualPreviewForSize, and OpenVisualPreviewForSize with VisualPreviewGrid, VisualPreviewFit, or VisualPreviewLarge to read, produce, or stream one specific retained recipe. Their maximum edges are 512, 2560, and 4096 pixels. Grid and fit leave the legacy active head alone. A newly produced large result becomes active. Reusing a cached result preserves a different head.

docbank.VisualPreviewSupportsMediaType(mediaType) checks normalized media types without reading or decoding a source. It reports whether a decoder path exists. Individual files can still return unsupported or failed results.

Inspect and repair stored content¶

Put and Create receipts include Physical. This field describes the raw, zstd, or packed representation on disk. Use it to inspect storage. Use ContentIdentity to identify document bytes.

Put is an idempotent content write, not an integrity repair primitive. Kit's structural dedup can reuse an existing canonical representation without hashing it, and the catalog can keep reading a packed copy even when a loose copy exists. When an application has trusted bytes for a known SHA-256 and size, RepairContent verifies the complete stream before replacing the physical copy the catalog reads. It preserves every node and historical version reference to that identity. Repairing packed content makes the catalog read a verified loose copy. A later repack reclaims the now-dead packed bytes.

Choose loose compression¶

New content remains loose until you call Pack. The standalone daemon selects zstd when a new loose object is at least 4 KiB and compression saves at least 10%. Embedded vaults keep raw loose storage by default. An owner can match the daemon policy or choose application-specific thresholds:

vault, err := docbank.New(ctx, docbank.Config{
    Root: root,
    LooseCompression: docbank.LooseCompressionOptions{
        Enabled:           true,
        MinBytes:          4 << 10,
        MinSavingsPercent: 10,
    },
})

Docbank keeps zstd only when the logical size meets MinBytes and the completed encoding saves at least MinSavingsPercent. Otherwise it publishes raw loose content. Enabling compression does not migrate or rewrite existing objects. RepairContent preserves an existing loose object's raw or zstd encoding. It applies this policy only when trusted bytes replace a packed or missing physical copy. The zero value writes new loose content as raw. Backup restore applies its own managed packing and compression policy, independently of these write options. Mixed raw, zstd, and packed content remains readable through the same verified API. Receipts report the chosen physical encoding and stored size without changing the logical SHA-256 or size. PutReceipt.Created reports a new logical node, while PhysicalCreated separately reports that the operation published new final loose authority.

An eligible write temporarily needs scratch space for both the raw object and its compressed candidate before Docbank chooses one to publish. LooseBacklog reports how much indexed loose content remains eligible for a pack pass, including both logical and physically stored bytes and a split between raw and compressed object counts. Applications can therefore schedule packing from physical storage growth without walking loose objects. The report does not make packing automatic.

Identify a vault and verify reads¶

vault.ID() returns the archive's UUID. JSONL backup and restore preserve that identity even when the restored vault has a different filesystem root, so applications can tell archives apart without treating paths as identity.

Do not trust an OpenContent stream until it reaches terminal io.EOF or Verify succeeds. Early Close does not drain the stream. Vault.Close waits for active operations and streams, closes storage, and releases the vault lock.

OpenContent and OpenVersionContent wrap ErrContentUnavailable when the catalog-authorized physical content cannot be opened or its physical size disagrees with metadata. Metadata lookup failures keep their existing ErrNotFound, ErrNotFile, or ErrClosed classification instead. A canceled physical open can match both context.Canceled and ErrContentUnavailable, so callers that distinguish cancellation should check the context error first.

OpenVersionContentRange selects a non-empty decoded logical byte range from one immutable version. Offsets and lengths address the same logical bytes described by ContentVersion.BlobHash and Size, whether the bytes are currently stored raw loose, zstd-compressed loose, packed, or in a secondary store:

part, err := vault.OpenVersionContentRange(ctx, versionID,
    docbank.ContentRangeOptions{Offset: 1 << 20, Length: 256 << 10},
)
if err != nil {
    return err
}
defer part.Reader.Close()
_, err = io.Copy(dst, part.Reader)

Range-read costs depend on storage:

• Raw loose content uses filesystem offsets.
• Compressed loose content is decoded to a temporary file first.
• Packed content is decoded in full into memory. Reading a small range can therefore need RAM for the whole decoded object.

A range outside the version's bytes returns ErrInvalidContentRange. The reader returns exactly the requested length, or io.ErrUnexpectedEOF if the stream ends early. It keeps the vault open until the caller calls Close.

A successful partial read checks the selected location, decoded size, and range bounds. It does not verify the whole object. Use OpenVersionContent to read and verify all bytes, or run maintenance verification.

Traverse and mutate the tree¶

Children reads the live virtual tree one page at a time, so a large directory is never loaded whole. Resolve a directory with Stat, then advance through its direct children with Limit and Offset:

manifests, err := vault.Stat(ctx, "/manifests")
if err != nil {
    return err
}
for offset := 0; ; {
    page, err := vault.Children(ctx, manifests.ID, docbank.ChildrenOptions{
        Limit:  500,
        Offset: offset,
    })
    if err != nil {
        return err
    }
    for _, child := range page.Items {
        // Inspect this bounded page.
        _ = child
    }
    offset += len(page.Items)
    if offset >= page.Total || len(page.Items) == 0 {
        break
    }
}

Pages contain directories first and files second, name-sorted within each kind. A zero limit uses DefaultChildrenLimit. One call cannot exceed MaxChildrenLimit. The total and page come from one metadata snapshot, but a caller that needs a complete stable traversal must avoid concurrent tree mutations between page calls.

Use Walk for a complete stable traversal. It pins one SQLite snapshot before returning and yields the selected root and its descendants in pages:

walker, err := vault.Walk(ctx, "/sessions", docbank.WalkOptions{PageSize: 500})
if err != nil {
    return err
}
defer walker.Close()

for {
    page, err := walker.Next(ctx)
    if err == io.EOF {
        break
    }
    if err != nil {
        return err
    }
    for _, entry := range page {
        // entry.Path is canonical within the pinned snapshot.
        _ = entry
    }
}

A zero page size uses DefaultWalkPageSize. No page can exceed MaxWalkPageSize. Paths are limited to MaxWalkPathBytes, and absolute tree depth is limited to MaxWalkDepth. Later tree changes do not enter the snapshot.

The walker loads the tree incrementally. Setup does not load the whole subtree. Each node needs at most two sibling range seeks and one child seek. The second sibling seek applies only when an include-trash walk finishes the node IDs sharing one path and advances to the next name.

Always call Walker.Close, including after io.EOF. Repeated calls are safe. It releases the read transaction, dedicated connection, and vault lease. Vault.Close waits for all walkers and content readers to close.

MovePath, TrashPath, and Restore return the resulting node and canonical path. Their optional positive IfRevision rejects stale mutations. IfRevision == 0 is unconditional. Restore accepts photo members inside trashed folders and recovers their group, those folders, and their other contents. EmptyTrash uses MaxRoots as a batch target, finishing complete photo groups even when they exceed it. Zero uses DefaultTrashEmptyMaxRoots, and More asks the owner to schedule another batch. Held counts roots the batch reached but kept because a connected photo member is live, too new, or retained.

Use BatchMove for an all-or-nothing reorganization of up to MaxBatchMoves nodes. Each source is either a path resolved inside the transaction or a node ID with the revision previously inspected. Every destination is the node's full final path, and its parent resolves in the planned final tree. An existing directory does not mean “move into.” Docbank validates the complete final tree before any change, so embedded applications can express file or directory swaps and nested moves without temporary names or partial completion.

Back up and restore an embedded vault¶

An application that owns a vault in-process can use the same portable, topology-independent backup format as the daemon:

repository, err := docbank.InitBackupRepository(backupRoot)
if err != nil {
    return err
}
snapshot, err := vault.CreateBackup(ctx, repository, docbank.BackupOptions{
    Tag: "before-import",
    Prepare: func(ctx context.Context) error {
        return snapshotApplicationCatalog(ctx, catalogSnapshot)
    },
    ExtraFiles: []docbank.BackupExtraFile{{
        Path: catalogSnapshot, RecordAs: "application/catalog.sqlite",
    }},
})
if err != nil {
    return err
}
proof, err := repository.Verify(ctx, docbank.BackupVerifyOptions{
    SnapshotID: snapshot.ID,
})
if err != nil {
    return err
}
if len(proof.Problems) != 0 {
    return fmt.Errorf("backup verification found %d problems", len(proof.Problems))
}
_, err = vault.RestoreBackup(ctx, repository, docbank.BackupRestoreOptions{
    SnapshotID:    snapshot.ID,
    Target:        restoreRoot,
    ProtectedRoots: applicationStorageRoots,
})
return err

Use OpenBackupRepository after process restart. Snapshots lists recovery points in chronological order. Save the repository's ID() and the returned snapshot's ID and MinReaderVersion with your recovery record. MinReaderVersion names the required backup format reader, not a Docbank release. Pass the saved snapshot ID to verification and restore so a newer backup on a shared repository cannot change which recovery point you use.

Capture prevents content reclamation for the whole snapshot. Ordinary appends resume once Docbank fixes the SQLite view that the backup will use. Physical maintenance waits until Docbank publishes the manifest.

To include your application's catalog in the same backup:

1. Use Prepare to create an immutable catalog snapshot while writes are paused.
2. Declare the file in ExtraFiles.
3. Keep the file unchanged until CreateBackup returns.

RecordAs is the file's relative location beneath the restored vault root. Your application must coordinate its own writes during Prepare. Docbank's freeze pauses Docbank mutations, not changes to an unrelated application database. The callback must not call vault mutations while that lock is held.

Extras larger than 64 MiB stream in chunks, including files over 4 GiB. Those snapshots require backup reader version 6. Once one exists, older readers cannot list snapshots, create backups, prune, or restore or verify the latest snapshot. Verification of all snapshots also fails. They can still restore or verify a supported older snapshot by ID. Upgrade every reader before writing large extras to a shared repository. Existing extras larger than 64 MiB are stored again as chunks on their first capture. Their old whole-file blobs remain until the older snapshots are removed and pruned.

Capture fails if an extra's length changes after it is opened. This does not detect in-place changes, so the immutable-source requirement still applies. Restore stages all extras as complete temporary files before replacing their destinations. Allow disk space for those copies alongside any existing files.

Mark files containing credentials or tokens as Sensitive. Docbank rejects sensitive files in a plaintext repository unless your application sets AllowPlaintextSecrets for that backup.

Restore always uses a separate root. It rejects overlap with the live vault or repository and checks content, SQLite integrity, and manifest statistics before making the result available. Supply other application storage in ProtectedRoots. Docbank checks those locations before restore cleanup too. Embedded restore builds a fresh primary store. It does not recreate secondary store placement.

If the original vault is unavailable, open the backup repository and restore directly without initializing or opening a source vault:

repository, err := docbank.OpenBackupRepository(backupRoot)
if err != nil {
    return err
}
_, err = repository.Restore(ctx, docbank.BackupRestoreOptions{
    SnapshotID: snapshotID,
    Target: restoreRoot,
    ProtectedRoots: applicationStorageRoots,
})
return err

An omitted snapshot ID selects the latest recovery point. Set SQLite in BackupRestoreOptions to use your application's driver, such as modernc.Driver{} from go.kenn.io/docbank/sqlite/modernc. Omitting it selects the build's default driver. An application using modernc in a CGO build should select it explicitly for restore too, because the default CGO driver needs the fts5 build tag. Restore includes declared host files along with vault content. It rejects the repository, declared protected roots, and targets held by another vault. Include any offline source or application storage you want to preserve in ProtectedRoots: the repository cannot infer their current locations. Vault.RestoreBackup also protects its open vault automatically and keeps that vault's configured SQLite driver.

Remove recovery points and reclaim storage¶

BackupRepository.Forget removes the snapshot records you select. BackupRepository.Prune reclaims backup storage that retained snapshots no longer need. Neither operation opens the source vault. Retention schedules and which recovery points to keep remain the embedding application's policy.

Preview each operation before committing to it:

selection, err := repository.Forget(ctx, docbank.BackupForgetOptions{
    SnapshotIDs: snapshotIDs,
    DryRun: true,
})
if err != nil {
    return err
}
// Present selection.Selected to the user. Call Forget again with DryRun false
// to apply the removal; selection.Forgotten is empty during a dry run.
fmt.Println("Selected recovery points:", selection.Selected)

cleanup, err := repository.Prune(ctx, docbank.BackupPruneOptions{DryRun: true})
if err != nil {
    return err
}
// After removing snapshots, preview cleanup again before calling Prune with
// DryRun false. PacksToRemove includes any old packs that will be rewritten.
fmt.Println("Packs selected for cleanup:", cleanup.PacksToRemove)

Forgetting alone does not reclaim packed bytes. It refuses to remove the last recovery point unless AllowEmpty is set (ErrBackupLastSnapshot), and refuses parents needed by retained incremental snapshots (ErrBackupSnapshotRequired). Both cleanup operations use Kit's exclusive repository lock, including dry runs. Contention returns ErrBackupRepositoryLocked. ForceUnlock is only for known abandoned locks.

Pruning removes wholly unused packs and rewrites packs with less than half their encoded payload still needed. Mostly-live packs retain unused bytes, so this is not full compaction or secure erasure. Planned byte counts cover old pack files and copied payload, not exact net savings or temporary-space needs. Inspect the returned report even when an error occurs: completed removals and writes may be partial. Interrupted pruning can be retried. These embedded operations do not add automatic retention or standalone CLI cleanup commands.

Maintain physical storage¶

Ordinary Put calls publish loose content. Call Pack when the embedded owner is ready to move authorized loose blobs into managed immutable packs:

report, err := vault.Pack(ctx, docbank.PackOptions{MaxBytes: 256 << 20})
if err != nil {
    return err
}
if report.More {
    // Run another bounded pass when scheduling allows.
}

MaxBytes is a soft committed raw-byte budget: the pass finishes the blob that crosses the budget, seals its pack, and stops. Zero is unlimited. The report includes packing, reconciliation, missing/corrupt content, and orphan cleanup outcomes. Embedded applications should surface those fields. A nil error alone is not a complete health report. Packing changes only physical representation. OpenContent keeps the same verified read contract.

Embedded GarbageCollect, Verify, and Repack calls are resumable bounded passes. WorkBudget.MaxObjects == 0 uses DefaultMaintenanceMaxObjects. Larger budgets must not exceed MaxMaintenanceObjects. A positive MaxBytes adds a soft byte bound, so one selected object may finish after crossing it. A zero byte bound is unlimited, but the object bound still limits each pass.

When a report has More, pass a non-empty NextCursor back in the same operation's next WorkBudget. Treat cursors as opaque and operation-specific. Malformed cursors and cursors from another operation return ErrInvalidMaintenanceCursor. A Repack pass can validly report more work with an empty cursor when completed mutations themselves reduce the candidate set. In that case, repeat it from an empty cursor. Cursors are continuation positions, not snapshot tokens: work inserted earlier in canonical order during a cycle waits for a later cycle started without a cursor.

The embedded calls do less than the standalone full-run commands. Embedded Verify checks a bounded page of blob bytes but does not perform whole-catalog metadata validation. Embedded GarbageCollect handles bounded unreachable catalog authority but does not enumerate untracked filesystem files. The daemon's verify and gc commands still run those full-run checks.

Your application decides when to empty trash and prune prior versions. GC can reclaim a blob only after no retained reference needs it. Embedded vaults have no background maintenance scheduler, so your application also chooses when to resume each bounded pass.

Choose SQLite¶

The build default preserves performance where CGO is available:

• CGO builds use github.com/mattn/go-sqlite3.
• CGO_ENABLED=0 builds use modernc.org/sqlite.

An application may select either adapter explicitly:

import (
    "go.kenn.io/docbank"
    "go.kenn.io/docbank/sqlite/modernc"
)

vault, err := docbank.New(ctx, docbank.Config{
    Root: root,
    SQLite: modernc.Driver{},
})

Use sqlite/mattn.Driver in a CGO build to select the CGO adapter explicitly. The adapter is selected when the vault opens. Query and transaction operations then run directly on that driver's database/sql pool. Standalone backup and restore paths use the same adapter boundary rather than silently switching SQLite implementations.

Ownership boundary¶

Do not point a daemon and an embedded application at the same root. New holds the same exclusive hierarchy lock as the daemon for the entire vault lifetime and fails if the requested root overlaps another active vault or restore target.

The standalone CLI remains daemon-first and never opens storage directly. Embedding is a distinct application ownership mode, not a second privileged path into a daemon-owned vault. External agents should continue to use the authenticated HTTP contract.