Wrapper

The batch-Job container (htrflow-batch, module htrflow_batch): fetch pages from IIIF, run the htrflow pipeline, stream per-page results to S3, publish the viewer manifest and the completion marker. This page is its whole contract — the Job it runs in, where its volume comes from, its environment, its exit codes and what it leaves behind. The narrative is in How it Works → The Wrapper; the failure semantics in Failure Handling. To run the wrapper without a cluster, see Quickstart.

The Job contract

A campaign is one Indexed Job, one index per volume: $JOB_COMPLETION_INDEX selects a line of the campaign's volumes.txt. Retries, the failure policy, the deadline and the grace period are in the rendered Job, field by field, in Rendered objects; admission is in Queueing.

Before the wrapper starts, the warmup-wait init container waits for /data/warmup/<pipeline-id>.done for up to warmup_wait_seconds. If the marker never appears it prints no warm-up marker at <path> after <n>s: the pipeline's warm-up Job has not finished and exits 13, which fails the index at once (Warm-ups fail the same way).

Where a volume comes from: volumes.txt

Every volume in a campaign becomes exactly one line of a ConfigMap — campaign-<name>'s volumes.txt — written by the converter from the campaign file's volumes: list, in file order. A two-volume render (one bare reference, one images: volume) produces this volumes.txt:

R0001203    https://<iiif-host>/<path>/R0001203/manifest
loose-scans images:https://example.org/scan1.jpg https://example.org/scan2.jpg

Line format (Volume.source_line() in models.py): <id> and the source, separated by the first tab — the shell prologue below splits on it. The source half is one of two shapes:

• a plain IIIF manifest URL — <id>\t<manifest-url> — for a bare reference (expanded through converter.yaml's source_template) or an explicit manifest: volume;
• images: followed by every URL under that volume's images: list, space-joined — <id>\timages:<url1> <url2> … — for an images: volume. Whitespace is the separator because a URL can never contain it, while a comma is legal anywhere in one (a IIIF size such as /full/2500,/), so validate refuses a URL with whitespace in it rather than encoding it. There is no manifest at all for these; the wrapper builds and publishes a synthetic one itself.

How index i reads line i. The campaign Job's container command (manifests/campaign-job.yaml) does the whole job in a few shell lines before execing the wrapper:

set -eu
mkdir -p "$HOME" "$TMPDIR" "$YOLO_CONFIG_DIR"
line=$(sed -n "$((JOB_COMPLETION_INDEX + 1))p" /campaign/volumes.txt)
[ -n "$line" ] || { echo "no volume for index $JOB_COMPLETION_INDEX" >&2; exit 13; }
id=${line%% *}; src=${line#*    }
export VOLUME_REF="$id"
case "$src" in images:*) export IMAGES="${src#images:}" ;; *) export IIIF_MANIFEST_URL="$src" ;; esac
exec python -m htrflow_batch

sed is 1-indexed, $JOB_COMPLETION_INDEX is 0-indexed, so index 0 reads line 1 (+ 1), and so on — index 0 above gets VOLUME_REF=R0001203 and IIIF_MANIFEST_URL=https://<iiif-host>/<path>/R0001203/manifest; index 1 gets VOLUME_REF=loose-scans and IMAGES=https://example.org/scan1.jpg https://example.org/scan2.jpg — the URLs are separated by a space, the one character a URL cannot carry unescaped. An index past the end of the file (completions is set from the same volume list, so it should not occur) gets an empty line and exits 13, FailIndex, rather than running the wrapper with nothing to work on.

Size limits.

• The ConfigMap: the API server refuses one over 1 MiB, so render splits a campaign into -part1, -part2, … at 10 000 volumes or 900 KiB of volumes.txt, whichever comes first.
• One environment string: IMAGES is one env value, and Linux caps one at 128 KiB (MAX_ARG_STRLEN). validate refuses an images: volume whose line is over 100 KiB. A campaign rendered before that rule could still fail with Argument list too long before the wrapper starts: split the volume, or give it a IIIF manifest.

Reading the file yourself. The ConfigMap is a normal cluster object:

$ kubectl get configmap campaign-<name> -n <namespace> \
    -o jsonpath='{.data.volumes\.txt}'
R0001203    https://<iiif-host>/<path>/R0001203/manifest
loose-scans images:https://example.org/scan1.jpg https://example.org/scan2.jpg

Environment contract

Source: packages/wrapper/src/htrflow_batch/config.py

Config.from_env fails fast (exit 13) with the full list of missing required vars, and when both or neither of IIIF_MANIFEST_URL and IMAGES are set. Config is a frozen pydantic model whose fields carry their own env name (Field(alias=…)) — there is no second table of names to keep in step, and the class-level default is the only default. A value pydantic cannot coerce (MAX_PAGES=abc) is also exit 13. The generated Configuration page lists these keys next to the other three surfaces'.

No wrapper setting is ever a secret. S3 credentials reach the pod as a mounted Secret file (AWS_SHARED_CREDENTIALS_FILE=/secrets/s3/credentials), never as an env var: an env var is readable in kubectl describe, in a crash dump and in every child process. test_config.py fails if any name below ever matches KEY|TOKEN|PASSWORD|SECRET_ACCESS.

Required:

Optional:

The per-volume wall-clock budget is not a wrapper setting: the campaign Job renders it as the pod's activeDeadlineSeconds (converter.yaml's max_seconds, or the pipeline's own). At the deadline the kubelet SIGTERMs the wrapper, which takes the 143 path below.

Results land at {S3_PREFIX}/{PIPELINE_ID}/{VOLUME_REF}/… (Config.volume_prefix).

In a deployment, a Job carries only the env its skeleton and converter.yaml give it, and with security.policies on, job-shape refuses one that carries more. Every setting above that neither sets — MAX_IMAGE_WIDTH, RESUME, the lookahead bounds, MAX_PAGES, the download settings, MAX_IMAGE_PIXELS, PAGE_TIMEOUT_SECONDS, LOG_SHIP_SECONDS — runs at its default there. The Set by column of Configuration says who can set each one.

Hand runs. Run by hand, outside a rendered Job, the wrapper needs the six required vars and a credentials source; the knobs worth touching are MAX_PAGES (cap it to 1 or a handful of pages, check the output, then run the real volume with MAX_PAGES=0), MAX_IMAGE_WIDTH, RESUME, MANIFEST_MAX_BYTES/FETCH_MAX_BYTES and LOG_SHIP_SECONDS. The compose stack does exactly this without a cluster — see Quickstart.

Workdir bound. LOOKAHEAD_BYTES bounds the images in WORKDIR_PATH to half the Job's memory-backed emptyDir: the default 1 GiB is half the default 2 Gi, and a pipeline at a named size gets LOOKAHEAD_BYTES rendered as half its size's workdir (Pod sizes). Heavier images shorten the lookahead window rather than overflow it (Memory bounds).

Warm-up entrypoint

python -m htrflow_batch.warmup runs in a pipeline's warm-up Job (htr-warmup-<id>): it builds the pipeline once, which downloads its models into HF_HOME, then drops the marker the batch pods' warmup-wait gate looks for. It does not use Config; it reads:

The warm-up Job also sets CUDA_VISIBLE_DEVICES="" and the same WORKDIR_PATH/HOME/TMPDIR/YOLO_CONFIG_DIR workdir paths as a batch pod.

Stages

config → setup → resume → load → stream → verify → publish; the current stage is what the termination log reports.

progress.json records the same stages except config, and adds two of its own: done after publish, and failed on any exit that is not a success. The file starts at setup, because the tracker that writes it is built only once the config stage has passed — so a ConfigError (exit 13: a missing or invalid environment) leaves no progress.json at all, and the termination message is the only evidence. Details in The Wrapper.

Exit codes

Exit 13 is raised for:

• a ConfigError: missing or invalid environment
• a manifest URL that is not http(s), or a manifest HTTP 400/401/403/404/410
• a manifest over MANIFEST_MAX_BYTES once decoded, or with a Content-Encoding other than gzip
• a manifest that is not a JSON object, has no canvases, or has a canvas with no image, a malformed shape or a non-http(s) image URL
• a bad pipeline: bad YAML, an unknown step or model class, a setting a step does not take, an Export step, or a pinned revision a key beside model_settings overrides (the last two before any model is built)

An exception that is also an OSError is never 13.

Exit 1 is raised for:

• a manifest 5xx, 429 or other status, a network error or the download deadline
• the verify gate: a page missing (neither uploaded nor recorded as failed), or a run where every page processed failed and nothing was resumed. The message lists missing and failed pages, and the errors of the first 10 failed ones
• any OSError, including a model missing from the offline cache (LocalEntryNotFoundError): a re-warm fixes it
• UploadOutage, after 5 consecutive S3 upload failures
• anything else

Exit 143 is the SIGTERM handler: termination log, final run-log ship, os._exit(143). A deadline kill also carries status.reason: DeadlineExceeded, which the read API shows as "error": "DeadlineExceeded". Pages already published are never redone.

A page that fails does not by itself fail the run. Failures write one structured reason to the termination log ({"stage": "stream", "permanent": false, "error": "verify failed: …"}), URL-redacted and clipped at 3500 characters with the counts and cause first. What each code costs is in Failure Handling; a dead htrflow worker thread is under A dead htrflow worker thread. How one page is fetched, retried or deferred is in From Image to Transcription.

The warm-up entrypoint uses the same codes and a {"stage": "warmup", "permanent", "error"} termination message:

• 13: a bad pipeline (as above, plus RepositoryNotFoundError and RevisionNotFoundError), HF_HUB_OFFLINE set, PIPELINE_PATH missing or unreadable, or a marker it cannot write. Its Job turns 13 into FailJob.
• 143: SIGTERM, including its own 3600 s pod deadline. Retried under backoffLimit: 2.
• 1: anything else, including a model not in the cache yet.

Modules

Source root: packages/wrapper/src/htrflow_batch/

Completion contract

manifest.json is written last, only after the verify gate confirms every page is accounted for — PAGE and ALTO in S3, skipped by resume, or recorded as failed with a reason — and its presence is the sole "done" signal for anything that lists results directly. A volume can therefore be done and have lost pages; pages_ok and pages_failed say which it was. It embeds the pipeline YAML and its sha256 (the drift ground truth), image_digest, per-page results, page_sources, page_source_digests (what a resume compares) and canvas_ids, the run metrics (wall_seconds, gpu_stall_seconds, pages_per_second, bytes_fetched) and viewer_url (field table).

On any failure the wrapper writes the termination log before exiting non-zero, never a completion marker. Every URL in the shipped run log, in termination messages and in page_sources is redacted (no userinfo, no query string; the boundary rules are in iiif.redact_urls). source_manifest in manifest.json is written verbatim — the manifest URL the Job fetched, or, for IMAGES volumes, the synthetic manifest id the wrapper published to sources/.

Live run log

<namespace>/status/logs/<pipeline>/<volume>.txt (under S3_PREFIX) is the run's own stdout/stderr, uploaded every LOG_SHIP_SECONDS while it changes and once more on exit, SIGTERM included, within a 90 s budget that fits the pod's 120 s grace period. The buffer keeps the first 1 MiB and the last 2 MiB. The warm-up does not ship a log. Details are in Events and signals.