Document understanding in Go¶

Use Docbank's Go packages to prepare documents for text search, OCR, or embeddings in your own application. OCR extracts text from document images. Embeddings represent text or media as numeric vectors for similarity search. Importing these packages does not start a daemon or open a vault.

Start with the provider-neutral contracts in document. Choose a provider only after deciding what bytes your application may send and what results it may retain. A rendition is a derived, readable representation of a document. Canonical evidence is the validated text and source locations behind it.

These are reusable Go contracts. Vault-owned publication, consent, backup, and search are described in Document processing. The daemon configuration reference separately lists its executable embedding adapters.

Choose a rendition provider¶

Verified page images documents the optional pinned local page runtime. Page geometry and images are separate from text renditions.

Each adapter implements an extraction contract with limits. “Operator-hosted” means you run the service and declare its destination. “Hosted” means the adapter calls an external provider. The format list and limits belong to the provider descriptor and profile, not to a filename extension.

Unstructured and Tika supply compatibility profiles for an operator's bridge. They do not install those services. Local-process providers also require the operator to supply the pinned runtime. Trafilatura on macOS or Windows requires a supplied, audited IsolatedRunner.

The rendition contract accepts an AuthorizedUpload bound to one inspected source. Providers cannot substitute a source URL for those bytes. Their receipts record the source, policy, provider identity, and usage. See the provider contract and rendition bridge schema for the normative types and wire format.

Extract EPUB locally¶

document/epub provides epub.in-process-v1 for one reflowable EPUB package with XHTML spine content and no META-INF/encryption.xml entry. Encrypted content and IDPF/Adobe font obfuscation are rejected as unsupported. Construct it with epub.New(epub.Profile{MaxDocumentBytes: 16 << 20, MaxUnits: 1000}). Both limits are required. Bytes range from 1 through 500 MiB, and virtual units range from 1 through 1,000,000. The descriptor reports local_process and binds both limits to its policy identity.

Pass the provider to an embedded vault through ProcessingProfileConfig.RenditionProvider, together with its processing profile. The daemon also selects this provider with the docbank-epub-rendition/v1 adapter. Its max_document_bytes and max_units must match the limits used to construct the descriptor. Use the public planning and consent flow before processing. Filename disclosure follows the profile's disclose_filename setting. Extraction works with the filename disclosed or withheld.

Each declared spine occurrence produces one ordered evidence unit with its archive path. Repeated references and linear=no entries count separately. Empty entries keep their location and consume zero virtual units. Head metadata and active XHTML content contribute no text. Images and visual layout have limited fidelity.

Usage counts the complete normalized Markdown with LF line endings. Each line wraps virtually at 80 Unicode code points, and 48 virtual lines make one unit. Every spine entry starts a new page. A 100-line chapter can therefore remain one evidence unit while consuming three virtual units. Wrapping leaves retained Markdown unchanged. Configure the processing profile's retained unit and text limits separately from the provider's virtual-unit ceiling.

The provider verifies the authorized source length and hash, applies the format detector's ZIP limits, and rejects unsupported packages with unsupported_input. Output, work, or virtual-unit overflow returns policy_rejected. A cumulative total exactly equal to MaxUnits is accepted. Truncated output and partially processed books produce no successful provider result.

Build canonical evidence and a rendition¶

Use NormalizeEvidenceV1 when an extractor can report source locations and structure. It validates SourceEvidenceV1 and produces NormalizedEvidenceV1 with stable identities for ordered units, regions, artifacts, and omissions. Locators distinguish pages, slides, sheets, records, messages, lines, media segments, and other document units. Media segments use half-open millisecond intervals: the start is included and the end is excluded. Adjacent and overlapping segments are valid, and an absent time remains absent. Completeness is reported as complete, partial, or degraded_provenance. Readable text alone does not prove complete source coverage.

BuildRenditionV1 derives sanitized Markdown, normalized units, and lexical segments from that evidence. Lexical segments are model-independent text spans for keyword search. Embedding chunks are built separately with the selected model's tokenizer. MarshalNormalizedEvidenceV1 produces the canonical retained evidence bytes and checksum.

The owning contracts are evidence, evidence validation and normalization, and renditions.

Normalize provider output¶

document.NormalizeDocument is the simpler text-normalization contract for ordered source units, such as pages. It turns those units into consistent text. The result includes heading paths, chunks, checksums, and source spans that locate each result in the original input. The same input and policy produce the same result.

policy, err := document.NewNormalizePolicy(2_000_000)
if err != nil {
    return err
}

normalized, err := document.NormalizeDocument(document.SourceDocument{
    Family:   "pdf",
    UnitKind: "page",
    Units: []document.SourceUnit{
        {Index: 0, Markdown: "# Quarterly report\n\nRevenue increased."},
    },
}, policy)
if err != nil {
    return err
}

The policy's structural values are fixed for its normalization version. The only caller-selected input is the maximum normalized document size. A future change to the algorithm or structural values requires a new normalization version, so existing checksums keep their meaning. Source family and unit-kind identifiers must be valid UTF-8 without control characters so they remain stable in fingerprints and provider-facing locators. Version 3 binds truncation into unit, chunk, and document checksums. A complete shorter document therefore cannot share an identity with a longer document whose retained evidence or remaining units were truncated by a normalization bound.

Build evidence from a supplied transcript¶

Applications that already have transcript text can pass it through the same evidence and rendition contracts as other document families:

evidencePolicy, err := document.NewEvidencePolicy(100_000)
if err != nil {
    return err
}
evidence, artifact, err := document.BuildTranscriptEvidenceV1(
    document.SuppliedTranscript{Provider: "beeper", Text: transcript}, evidencePolicy,
)
if err != nil {
    return err
}
_ = artifact // retain the provider transcript with the source record

Use a lowercase identifier for the provider. The supplied-transcript/v1 JSON artifact keeps the exact provider name and transcript text. Normalized evidence and the rendition (the derived document result) still apply their Unicode, Markdown, and character limits.

The evidence uses the audio family, a generic unit, and degraded_provenance completeness. Supplied text has no timing or speaker information. Your application must associate it with the audio source and source version. This function does not inspect audio, run speech recognition, verify the provider's identity, or create ingestion and search records.

An application that wants the same transcript to travel through the rendition contract, sealed upload, authorization, receipt, and retained artifact can construct a document/suppliedtranscript provider whose Source uses the sealed audio digest as its key. The provider identity includes the caller's source binding. Use a different binding for each transcript source to give it a distinct descriptor. Docbank cannot verify that the binding identifies the Source implementation or the transcripts it returns.

Use document/mediatranscript when the source supplies segment timing. Its media-transcript/v1 artifact retains ordered text, start and end milliseconds, and an optional speaker for each segment. ParseSubRip accepts the bounded subrip/v1 grammar, keeps styling tags verbatim, and infers nothing about language, speakers, or timing. Build converts that artifact into audio or video source evidence and a retained transcript artifact. Segment starts must not regress, but adjacent and overlapping cues remain distinct. Speaker text participates in evidence identity when present. Omitting it keeps the supplied form free of invented values.

Supplied caption timing limits¶

The supplied-captions profile rejects the entire caption file if any cue ends after the recording's measured duration, even by one millisecond. It preserves the supplied times without rounding, clipping, or adding a tolerance. Check that the captions match that recording before importing them.

Run Mistral OCR safely¶

Mistral uploads are refused until an operator supplies a validated capability manifest for the configured endpoint, model, and policy. A capability manifest records what a live provider probe demonstrated. Docbank supplies no live manifest. Provider documentation alone does not authorize uploads.

The operator prepares that evidence in this order:

1. Call WriteProbeFixtures to create synthetic test documents.
2. Call ValidateProbeFixtures locally, without credentials or HTTP.
3. Create NewClient(API key) and call RunCapabilityProbe.
4. Review and retain the resulting CapabilityManifest.

Fixture generation creates the candidate formats deterministically. Five legacy formats require operator-supplied synthetic seeds named doc, ppt, xls, numbers, and msg. Fixture and staging directories must be private. The authenticated probe authorizes PDF through a provider-request bound. It can authorize PPTX when its local slide count matches the provider's processed units. It records primary-fixture evidence for TXT, Markdown, CSV, JSON, JSONL, YAML, Go, Python, JavaScript, RST, LaTeX, XML, EML, and MSG after provider acceptance. Native XLSX remains unauthorized through the capability probe, even when extraction succeeds during a probe. Configured render policies use the PDF authority for XLSX instead.

Text formats use provider_response enforcement. Docbank does not count lines, records, or messages as pages. It checks that Mistral returns at least one page, that the number of returned pages equals pages_processed, and that both stay within MaxUnits. A CSV with more than 5,000 rows can therefore be submitted when it fits the input byte limits.

Before upload, MaxDocumentBytes limits every file. Text detection also rejects text inputs above 50 MiB. These byte limits do not establish a provider page or spending limit. Mistral may process and charge for a document before Docbank rejects an over-limit response. Applications that require a pre-upload page or spending bound must not authorize text formats through this route.

Capability manifests use schema v4. Existing v3 manifests are rejected for all formats, including PDF and PPTX. Rerun the authenticated capability probe and replace the manifest before processing documents. Also rerun the probe if validation reports that a registered format "does not explain its unverified bound". Review application consent against the resulting policy fingerprint.

DOCX, DOC, ODT, RTF, PPT, XLS, ODS, and XLSX share one production route. Set PolicyConfig.RenderPDF to a validated renderpdf.Policy and supply PDF capability evidence in the same manifest. Policy.Authorize keeps the original format identity but grants a local-exact bound for the render route. Native office probe results remain diagnostic observations and do not authorize native office bytes.

For each production document:

1. Call Policy.Authorize(validated manifest, declared format).
2. Check your application's consent record against PolicyFingerprint.
3. Call Prepare with private staging storage.
4. Call Process. The detected format must match the authorization.
5. Call Release on success or failure.
6. Call NormalizeDocument(Result.Document, Policy.NormalizePolicy()) on a successful result.

Prepare copies one input into a private, bounded, immutable staging file. For direct formats, Process reopens and verifies those bytes for every attempt. It derives request options from the policy and authorization, bounds the response, and converts validated provider output into document.SourceDocument. For a configured render profile, it calls renderpdf.Convert once, counts the generated PDF, checks MaxUnits, MaxPages, MaxPDFBytes, and MaxDocumentBytes before HTTP, and uploads that PDF on every retry. The returned document keeps the original family and page unit kind. Its result carries the original and generated PDF hashes. Call Release on every success or failure path.

To display a local preparation failure, follow Show preparation failures. Keep the original error for retry and cancellation decisions.

The rendition adapter verifies source identity before submission. For PDFs it compares the returned page count with that inspected count. For PPTX it counts the listed PresentationML slides, rejects invalid slide references or over-limit decks before upload, and compares the provider's processed count with that local count. Configured office formats use the shared render route. Rendered Word keeps complete page evidence. Rendered presentations and spreadsheets use one generic degraded evidence unit with an omission because PDF pages do not identify the original units. The receipt records the generated PDF hash and input bytes. Text formats use the response checks described above. See Mistral rendition processing for the source and result checks.

The PPTX count includes hidden slides. This assumes Mistral processes every listed slide. The probe fixture contains one visible slide and does not verify hidden-slide behavior. If Mistral skips hidden slides, the count comparison fails after upload and may incur provider charges.

Native XLSX production uploads remain blocked because its native provider authority is unverified. With PolicyConfig.RenderPDF, XLSX follows the render route and Mistral enforces the generated PDF page count before upload. The returned processor result keeps the original family and page unit kind. Rendition evidence for rendered presentations and spreadsheets uses generic degraded provenance.

The importing application remains responsible for credentials, human consent, spending and scheduling limits, durable manifests, job orchestration, persistence, and search serving. Those application decisions are outside the reusable packages and their policy identity.

Prepare text for semantic retrieval¶

document.BuildEmbeddingInputs converts canonical normalized evidence into a sealed generation of provider-ready document inputs. It never reads the retained Markdown, so YAML frontmatter, checksums, build identifiers, and navigation metadata cannot enter embedding text.

The processing profile's EmbeddingChunkPolicyV1 declares how to build chunks. It records the tokenizer name and revision, content token budget, overlap, formatter, truncation policy, and attachment-context rules. A tokenizer determines the token units that the model counts.

document.NewInputPolicy combines that declaration with:

• A document.Tokenizer implementation reporting the declared identity.
• The binding's fixed model-input contract.
• The lexical evidence fingerprint, which identifies the text evidence.
• The provider's hard token and byte limits for the complete rendered input.

The resulting policy belongs to one rendition-chunk binding.

policy, err := document.NewInputPolicy(binding, tokenizer, lexicalFingerprint, nil)
if err != nil {
    return err
}

generation, err := document.BuildEmbeddingInputs(evidence, policy, document.GenerationLimits{
    MaxInputs:              10_000,
    MaxTotalContentTokens:  4_000_000,
    MaxTotalRenderedTokens: 5_000_000,
    MaxTotalContentBytes:   64 << 20,
    MaxTotalRenderedBytes:  80 << 20,
    MaxFittingWorkTokens:   50_000_000,
    MaxFittingWorkBytes:    1 << 30,
})
if err != nil {
    return err
}

Page, heading, region, and table boundaries are preferred before token splitting. The configured token budget applies to content only. The document role envelope and any declared attachment context are rendered on top of it, and the complete rendered input is re-counted against the provider limits. Overlap is measured in emitted tokens, and every input keeps its heading path and the source span it was reconstructed from.

Generation limits can reject a build but cannot change its output. They do not enter the fingerprints. All other policy values enter the generation's policy fingerprint. If declared, AttachmentContextSnapshot also binds the attachment titles and context to the generation's identity.

Use MarshalEmbeddingInputGeneration to produce the canonical byte form. DecodeEmbeddingInputGeneration accepts only that form, under limits the caller supplies. It rejects forged totals, policy fingerprints, and checksums. ToEmbeddingInputs accepts only the model-input contract used to build the generation.

EgressIdentity gives applications separate endpoint-sensitive fingerprints for document embedding and query embedding. Credentials are not part of those identities. VectorSpaceIdentity separately pins provider, model revision, dimension, and normalization without an endpoint. An application can then reuse compatible vectors while still requiring fresh consent when their destination changes.

The shared retrieval helpers make omitted and auto search lexical, so a query is not sent to an embedding provider unless the caller selects semantic or hybrid mode. Candidate limits default to 100 and cannot exceed 1,000. CollectScopedCandidates requires the backend to apply scope before its vector cutoff and pages until authoritative exhaustion or a limit+1 overflow probe. Reciprocal-rank fusion preserves lexical and semantic signals and reports overflow from either input lane or the fused union.

document/embedding/eval runs versioned public or synthetic corpora through repeatable retrieval systems. Reports include Recall@5/10/20, nDCG@10, MRR, hit@1, hit@10, critical misses, provider calls and input/output usage, query latency, token evidence, and cost evidence. A system can add an optional reranker over a top-N prefix. Ties keep their retrieval order and the remaining results stay in place. Repeated trials retain query observations and report empirical minimum, mean, maximum, and nearest-rank p95 values. The developer comparison uses a temporary real Docbank store and the existing lexical, vector-index, and fusion owners. Its synthetic documents are only test data. Offline provider rows and synthetic vector rows validate wiring only, so they do not support a quality recommendation. Missing provider runs or usage evidence remain unavailable. Skipped rerank calls add no usage or pricing basis. Cost totals use only observations from stages that ran. Applications should keep raw as the default until measured results justify a different recipe.

To run the live comparison, set COHERE_API_KEY and TYPESAFE_API_KEY, then run RERANK_EVAL_LIVE=1 go test -tags fts5 ./scripts/rerank-eval -run '^TestLiveRerankComparison$' -count=1 -v from the repository root. Ordinary tests compile the live adapter setup and check pricing inputs without making live calls. To include costs, also set RERANK_PRICING_DATE (YYYY-MM-DD), TYPESAFE_MICROS_PER_TOKEN, and COHERE_MICROS_PER_SEARCH_UNIT together.

Choose an embedding provider¶

The adapters below implement document.EmbeddingProvider. Their descriptors pin the model-input contract, input kind, vector dimensions, and compatibility identity. A provider package is not a daemon configuration option. Applications must supply matching profiles, authorization, and named credential resolution.

openaicompat.BGEM3Profile and openaicompat.Qwen3Profile build reviewed self-hosted profiles. BGE-M3 uses one dense 1,024-dimensional vector. Qwen3 supports its 0.6B, 4B, and 8B models at their native dimensions and requires a query instruction. Both pin weights and tokenizer revisions, pooling, and sequence limits. These profiles exclude sparse and multi-vector outputs. See the deployment contracts.

The existing custom text contract also supports an optional EmbeddingGemma 2 native-768 recipe. Its configuration and synthetic transport checks do not qualify a live model deployment. The operator supplies the server and its immutable recipe.

Voyage contextual requests contain one document's ordered chunks. The shared Voyage and Mistral adapters mark their mutable hosted aliases as export-only and do not advertise a serving-time text query encoder. Do not infer query compatibility from an equal vector dimension.

Gemini's profile selects inline bytes or the Files API. Direct-file requests need matching inspected capability and disclosure fingerprints. The Files API path includes bounded polling and cleanup. Its policy identity records the provider retention ceiling. The direct-file adapter accepts PNG/JPEG images, WAV/MP3 audio up to three minutes, MP4/QuickTime video up to two minutes, and PDFs up to six pages, subject to matching inspection bounds. The Gemini contract separates that provider retention from Docbank's local derivative retention.

The standard embedding bridge sends a canonical manifest before any file parts and validates the synchronous response against the input identities. Its OpenAPI contract and JSON schemas own the wire limits.

Detect and bound visual attachments¶

media.Detect identifies JPEG, PNG, WebP, GIF, and MP4 from their bytes. It does not trust the declared media type or decode pixels and samples. It reads the metadata needed to enforce input limits:

• Dimensions, including every declared axis bound for MP4 and every GIF frame within the logical screen.
• Verified frame counts, including animated WebP and APNG.
• MP4 duration, using the longest movie, track, media, or sample-timing value.

MP4 video tracks must contain out-of-band H.264 (avc1/avcC) or H.265 (hvc1/hvcC) dimensions, plus mdhd and stts timing. The detector rejects in-band or missing evidence as malformed. It does not substitute a smaller container summary.

media.Evaluate checks byte, pixel, frame, and duration caps, plus the policy's still-image, animation, and video settings. It returns stable reasons such as too_many_pixels or animated_not_allowed. If the policy caps duration, video with an unmeasurable duration is refused.

metadata, reason, err := media.Inspect(reader, size, "image/png", media.DefaultPolicy())
if err != nil {
    return err
}
if reason != media.ReasonEligible {
    return recordIneligible(reason)
}

The package has no notion of attachment ownership, roles, or hashes. Those are application pre-filters.

Embed images and video with Voyage¶

Voyage uploads fail closed until an operator has produced and supplied a validated capability manifest for the pinned endpoint, model, and dimension. Docbank ships no live manifest. Each capability is authorized separately: JPEG, PNG, WebP, still GIF, animated GIF, and MP4 documents; text queries; JPEG, PNG, WebP, and still GIF image queries; text-then-PNG queries; text-then-media documents probed per format; and mixed-format batches at the policy limit. Recorded probe evidence decides whether animated GIF, video, or a given query shape may be sent.

The operator prepares a manifest in this order:

1. Call WriteProbeFixtures to generate JPEG, PNG, and GIF fixtures and copy operator-supplied WebP and MP4 seeds.
2. Call ValidateProbeFixtures locally, without credentials or HTTP.
3. Create NewClient(API key) and call RunCapabilityProbe.
4. Review and retain the resulting CapabilityManifest.

The Go standard library cannot encode WebP or MP4, so the operator supplies synthetic seeds named image_webp.webp and video_mp4.mp4. The operator also supplies image_webp_alt.webp and video_mp4_alt.mp4: variants of the same format with different content, so that pixel contribution is demonstrated within each format. The seed directory, destination parent, and published fixture directory must be owner-private. Fixture generation publishes a complete new directory atomically instead of updating an existing directory. The probe stores no media, vectors, or responses. The manifest records only sanitized pass, reject, and fail observations.

For each production item:

1. Call Policy.AuthorizeAll(validated manifest).
2. Check your application's consent record against PolicyFingerprint.
3. Call media.Inspect and require the item to be eligible under the policy.
4. Call Client.EmbedDocuments(inputs, authorizations).

EmbedDocuments accepts only the probed input shapes [media] and [text, media]. Before allocating a request, it enforces item and byte limits. It also:

• Detects every media part again and serializes from the detected metadata.
• Requires authorization for each part's format and animation state.
• Requires that format's interleaved capability for [text, media].
• Requires the batch capability for more than one input.

EmbedQuery accepts [text], [image], or [text, image]. Each shape needs its own probed capability. Image authorization is per format. Combined text and image queries support PNG only.

Every response vector must have a unique index and the exact dimension. The client does not follow redirects. Provider response bodies do not appear in errors. Use IsRetryable, RetryAfter, and MetricsFromError when scheduling retries.

The importing application remains responsible for credentials, human consent, spending and scheduling limits, durable manifests, job orchestration, vector storage, and search serving. Importing this library does not enable a vault workflow. The daemon's separate retained-job requirements are documented in Configuration.

Convert CSV locally for PDF OCR¶

Use document/csvpdf to convert a declared text/csv source into a PDF for the existing PDF OCR path. Conversion runs locally and never uploads content or looks up credentials.

The converter verifies the source byte count and SHA-256, parses records, labels each cell, and counts PDF pages before returning a result. It closes ocr.Source.Content on every success and failure path.

policy, err := csvpdf.NewPolicy(csvpdf.DefaultLimits())
if err != nil {
    return err
}
converted, err := csvpdf.Convert(ctx, originalCSV, policy)
if err != nil {
    return err
}
receipt := converted.Receipt()
pdfSource, err := converted.Source()
if err != nil {
    return err
}
// Keep the receipt with pdfSource before a later caller authorizes upload.

If conversion fails, follow Show preparation failures to choose a display message.

The defaults are also hard ceilings. NewPolicy accepts positive, tighter limits.

The embedded Go Mono font covers a subset of Latin, Greek, Cyrillic, and common punctuation or symbols. Conversion rejects missing glyphs, characters outside the Unicode basic multilingual plane, combining marks, shaping scripts, formatting controls, and control characters other than cell newlines. Use precomposed accented characters such as é.

CSV quoting, blank lines, and CRLF normalization follow Go's encoding/csv. The output preserves empty cells, variable-width records, and quoted newlines. Long cells continue onto later pages. Formulas remain literal text.

Keep the receipt with the source. It records original and generated hashes and byte counts, the converter version, the policy fingerprint, and the mapping from generated pages to original records and cells. Span identifiers start at one. A record may span several pages. To cite a CSV cell from PDF OCR output, use this mapping to translate the generated page reference.

The receipt does not authorize upload. Your application must:

1. Verify the PDF capability manifest and consent.
2. Choose PDF byte and page limits large enough for the conversion.
3. Retain the conversion policy with that consent.

This Go API does not add CSV OCR to the daemon or CLI.

Show preparation failures¶

mistral.Prepare and csvpdf.Convert attach ocr.PreparationError to local source and conversion failures. Extract it with errors.AsType[*ocr.PreparationError] and display its Error() value. The description contains fixed text. The original cause remains available through Unwrap() for private diagnostics.

Never display the original err.Error() or an unwrapped cause. The full error may include joined cleanup errors with filesystem paths or parser details, even when it contains a PreparationError. If none is present, use a fixed fallback message.

Check cancellation and deadlines first. A canceled operation may also contain a description for a source-close failure:

func preparationMessage(err error) string {
    switch {
    case errors.Is(err, context.Canceled):
        return "Document preparation canceled"
    case errors.Is(err, context.DeadlineExceeded):
        return "Document preparation timed out"
    }
    if description, ok := errors.AsType[*ocr.PreparationError](err); ok {
        return description.Error()
    }
    return "Document preparation failed"
}

Use the original error for scheduling. For ocr.Processor failures, use ocr.ErrorKindOf(err). For direct mistral.Prepare calls, use errors.Is with mistral.ErrSpoolCapacity, mistral.ErrSpoolUnavailable, and mistral.ErrInvalidSource. Use errors.Is for context errors in either path and for csvpdf.Convert. Display descriptions are not retry classifications. Do not match their text to decide whether to retry.

Convert office files locally to PDF¶

Use document/renderpdf when an application needs a local PDF derived from DOCX, DOC, ODT, RTF, PPT, XLS, ODS, or XLSX. The converter uses an operator-pinned LibreOffice executable. It first writes FODT, FODP, or FODS inside the Linux sandbox, checks the normalized XML for external or active content, and then renders only admitted bytes to PDF.

Set up the renderer¶

The built-in runner targets Linux on amd64 and arm64. It requires user, mount, PID, network, and IPC namespaces, seccomp filters, and Landlock filesystem ABI 3 or newer. The application must be allowed to create those namespaces. On Ubuntu, AppArmor may block unprivileged user namespaces. The operator's AppArmor policy must allow the launcher to create them. Container policy must also permit the required namespaces. Conversion fails if the required isolation cannot be installed. macOS and Windows require an application-supplied audited renderpdf.Runner.

On Debian or Ubuntu, install LibreOffice Writer and record the approved executable's SHA-256 during operator setup:

sudo apt-get update
sudo apt-get install --no-install-recommends -y libreoffice-writer libreoffice-impress libreoffice-calc
sha256sum /usr/lib/libreoffice/program/soffice.bin

DefaultRuntimeRoots selects Debian/Ubuntu paths for amd64 and arm64, including LibreOffice, its libraries, and fonts. Other layouts need an explicit list of runtime roots. Discover and retain the manifest during operator setup, then build a policy from its files, symlinks, and identity. Pass the approved executable digest as executableSHA256:

package renderer

import "go.kenn.io/docbank/document/renderpdf"

func newOfficeRenderPolicy(executableSHA256 string) (renderpdf.Policy, error) {
    manifest, err := renderpdf.DiscoverRuntime(renderpdf.DefaultRuntimeRoots())
    if err != nil {
        return renderpdf.Policy{}, err
    }
    return renderpdf.NewPolicy(renderpdf.Renderer{
        Executable:       "/usr/lib/libreoffice/program/soffice.bin",
        ExecutableSHA256: executableSHA256,
        Runtime:          manifest.Files,
        RuntimeSymlinks:  manifest.Symlinks,
        RuntimeIdentity:  manifest.Identity,
    }, renderpdf.DefaultLimits())
}

Keep the policy for subsequent conversions. Changed runtime bytes or a changed executable fail verification. Approve the updated installation and rebuild the policy after a LibreOffice, library, or font update. Do not rediscover the runtime or replace the approved digest for each input document.

Convert a source¶

converted, err := renderpdf.Convert(ctx, source, "docx", policy)
if err != nil {
    return err
}
receipt := converted.Receipt()
pdfSource, err := converted.Source()
if err != nil {
    return err
}
// Keep receipt with pdfSource before a later caller authorizes upload.

The DOCX, DOC, ODT, and RTF profiles emit FODT. PPT emits FODP. XLS, ODS, and XLSX emit FODS. Normalization strips unsafe external and active constructs where LibreOffice can remove them. The normalized scan rejects any such constructs that survive. It rejects every xml:base attribute, so local fragment links cannot inherit an external base URI. Local formulas, internal fragment links, and embedded raster images remain valid. Ordinary hyperlinks to external sites or relative files are rejected if they survive normalization. Embedded OLE objects, plugins, applets, and nested documents are also rejected. The scanner admits two LibreOffice metadata forms: an empty Basic script container whose library list is empty or contains only the Standard embedded-library marker, and presentation layout placeholders for all ODF object types. Local spreadsheet filter and sort ranges are also admitted. The scanner still rejects script code, event handlers, external or relative links, DDE, external formulas, database sources, charts, OLE, plugins, embedded documents, and other active objects. The render policy limits normalized bytes, PDF bytes, work bytes, XML depth and elements, and generated PDF pages through MaxPages.

Each XML character-data token, including base64 image data, is limited to 1 MiB. A document within the overall byte limit can still exceed this limit when it contains a large embedded image.

The receipt records both source and normalized identities, the PDF hash, the page count, the policy, and the runtime identities. Result.Source is a fresh application/pdf source. The receipt does not authorize upload. The Mistral adapter reuses this receipt for its configured office route. No daemon option or CLI operation is added. Native office authority remains separate when the render policy is absent.

Package boundary¶

Dependencies run in one direction:

application storage and workers
  -> document/mistral (optional provider transport)
  -> document (provider-neutral normalization)

application storage and workers
  -> document/voyage (optional provider transport)
  -> document/media (provider-neutral detection and eligibility)

application storage and workers
  -> document/embedding (provider-neutral text planning and retrieval contracts)
  -> document (provider-neutral normalization)

No public package imports Docbank's vault, database, daemon, or queue. Local extractors can use document without depending on Mistral or any network transport. Applications can use document/embedding without a database or provider client, and can use document/media alone to record image dimensions at ingest without any provider.