Dev cluster

A disposable single-node cluster with one NVIDIA GPU, with everything the platform needs from outside included: charts/htrflow-devstack provides RustFS as the S3 store, an image registry in its own registry namespace, and the NVIDIA RuntimeClass and device plugin, next to one release of charts/htrflow-batch. It uses the published images, so nothing has to be built. None of it is production-shaped: never install it next to real data or real credentials (Security lists the caveats); Deploy is the real install.

Install the stack

You need a Kubernetes cluster with one NVIDIA GPU node, a kubeconfig for it, and git, kubectl, helm, make and uv.

git clone https://github.com/AI-Riksarkivet/htrflow-batch && cd htrflow-batch
make install            # uv workspace: the converter CLI (htrflow-campaigns)
make install-kueue      # Kueue: the queue and GPU quota
make install-devstack   # Kyverno, then the devstack chart: S3, registry, device plugin

The cluster targets take the release name and namespace from HTR_RELEASE and HTR_NAMESPACE (.env, defaults htr and htr-batch); <namespace> below is that namespace. make install-devstack takes KYVERNO=false to skip Kyverno and NVIDIA_DEVICE_PLUGIN=false for a cluster that already has the device plugin (Gotchas).

kubectl -n <namespace> create secret generic htr-session \
  --from-literal=key="$(openssl rand -base64 32)"
helm upgrade --install htr charts/htrflow-batch -n <namespace> \
  --set resultsUrl=http://<node-address>:30800/results \
  --set results.sessionSecret=htr-session \
  --set results.keyDerivation=none \
  --set network.apiServer.cidr=<apiserver-address>/32 \
  --set network.iiifCidrs='{<iiif-source-cidr>}' \
  --set network.clusterCidrs='{<pod-cidr>,<service-cidr>}' \
  --set network.web.allowPublicIngress=true \
  --set security.policies.allowDisabled=true
make psa-labels

• <node-address> is where your browser reaches the node: the web front on port 30800, and the results through it under /results. Not directly reachable? See Reaching it from a workstation.
• htr-session holds the key the results proxy seals login sessions with. results.keyDerivation=none says RustFS issues S3 keys: the login form takes the access key as the user name and the secret key as the password.
• <apiserver-address> is the API server as pods reach it; on one node, usually the node's address.
• network.iiifCidrs must cover the host you transcribe from; campaign pods reach nothing else.
• network.web.allowPublicIngress accepts every client address, and security.policies.allowDisabled accepts the Kyverno policies off. Both are right for a disposable cluster only. To turn the policies on, add --set security.policies.enabled=true and --set security.allowedImageRepos='{docker.io/riksarkivet/,rustfs/}' (plus <registry>/ for your own images).

Then write a campaigns repo and apply it:

uv run htrflow-campaigns init my-campaigns

In my-campaigns/converter.yaml set results_url to the same URL as resultsUrl, and list your volumes in campaigns/demo.yaml (Run a campaign). Then:

make campaigns-apply DIR=my-campaigns

http://<node-address>:30800/ shows the campaign as it runs. Log in as htr-reader, the read-only RustFS user the devstack created, with the password from its Secret: kubectl -n <namespace> get secret htr-results-login -o jsonpath='{.data.password}' | base64 -d. The RustFS root keys, which can write, are in the S3 Secret htr-batch-s3.

Check resume

Once a few ALTO files exist under <namespace>/<pipeline>/<volume>/alto/ in the bucket, force-delete the running campaign pod:

kubectl -n <namespace> delete pod <pod> --force --grace-period=0

The retry pod's log says [<volume>] resume: <n> done, <m> to process. It does not redo the published pages, and the index still reaches Complete.

.env

Cluster constants for the Makefile and the compose stack live in a git-ignored .env at the repository root. .env.example holds the defaults and is loaded first, so a key missing from .env keeps its default. Copy it and adjust:

The chart takes the same values through --set or a values file; the Python packages take theirs from their own environment (Configuration).

The GPU wrapper image

The wrapper must be built for the node's own architecture. An image built for another architecture runs on the node only under CPU emulation, where the GPU is unreachable and uv crashes, so the build always runs natively on a machine of the node's architecture (Releasing).

The wrapper dockerfile builds its htrflow base itself, from the htrflow commit it pins, so one target is enough on any architecture:

make poc-push          # build-wrapper + build-web for the host's architecture, push both, print digests

To try an htrflow change before it is pinned, build from a local checkout instead; it replaces the pinned source and nothing is written into it:

make poc-push HTRFLOW_SRC=<htrflow-checkout>

The image then stamps the checkout's git describe --tags --always --dirty as HTRFLOW_BASE_REVISION, because manifest.json only knows htrflow's package version. The checkout's pyproject.toml must still be the one the committed lock in .docker/htrflow-base/ was made for; the build refuses it otherwise rather than resolve afresh. After moving the pinned commit (HTRFLOW_REF in the wrapper dockerfile), refresh the lock and review the diff:

make lock-htrflow-base          # UV_LOCK_ARGS=--upgrade moves every pin

poc-push prints the wrapper and web digests. The wrapper's goes into pipelines/<id>.yaml in the campaigns repo, the web front's into web.image. While iterating, security.allowTagImages=true accepts a tag (IMAGE_TAG, default dev) instead of a digest; a tag is then pulled on every rollout.

Rebuilding the web image

The campaign browser and the Universal Viewer are baked into the web image next to the read API, so any UI change needs a rebuild and a new web.image:

make build-web                              # bun-builds the SPA, clones + patches the viewer, tags <registry>/htrflow-web:<tag>
docker push <registry>/htrflow-web:<tag>    # prints the digest to pin as web.image
dagger call build-web                       # the same dockerfile, through the dagger engine

The image runs unprivileged and listens on port 8081; the chart's containerPort and Service targetPort follow it, and web.nodePort is the outside port. Then upgrade the release with the new digest.

The in-cluster registry

The devstack registry is a NodePort Service registry (port 5000) in the registry namespace. It is unauthenticated and serves plain HTTP, so:

• the node's container runtime must be allowed to pull from <registry> without TLS — runtimes typically allow that for a loopback address and need explicit registry configuration for any other;
• docker push from the node to <registry> needs the same allowance in the Docker daemon, or a loopback address;
• from a workstation, kubectl -n registry port-forward svc/registry <local-port>:5000 gives you a push address.

The registry does no garbage collection of its own. To reclaim space, delete tags over the registry API, then run registry garbage-collect /etc/distribution/config.yml in the registry pod. Pipeline files and web.image pin <registry>/…@sha256:<digest> references, which resolve only on this cluster.

Applying a campaigns repo

Because its pipeline files pin digests in <registry>, a dev cluster's campaigns repo belongs on its own branch or in its own repository, never on a shared main. Nothing in the cluster clones it; render and apply it from a checkout with a kubeconfig for the cluster:

export KUBECONFIG=<kubeconfig>
make campaigns-apply DIR=<campaigns-repo>           # render + apply, pipelines first
make campaigns-apply DIR=<campaigns-repo> PRUNE=1   # also cancel what this render no longer contains
make e2e DIR=<campaigns-repo>                       # validate, apply, wait for every campaign Job to finish

campaigns-apply runs htrflow-campaigns apply <campaigns-repo> --out <campaigns-repo>/rendered --namespace <namespace> (CLI); it is safe to re-run. Repeat after every commit to that branch — nothing watches it for you. PRUNE=1 cancels everything the checkout does not contain, so only run it against the whole repo. make e2e waits for the warm-up Jobs, then polls every campaign Job until it is Complete or Failed (CAMPAIGN_TIMEOUT seconds, default 3600). It fails if any Job ends Failed, if no campaign Job exists, or if kubectl cannot read one. Otherwise it finally requests /api/v1/jobs from the web front's NodePort on the machine running make. Editing an already-rendered campaign's volume list in place hits the append-only rule (Campaign & Pipeline YAML).

Addresses a pod fetches

Anything you put in a campaign file — a fixture manifest on the RustFS bucket, say — must use the in-cluster form (http://rustfs.<namespace>.svc.cluster.local:9000/…), because the pod, not your browser, fetches it.

Reaching it from a workstation

The browser needs one address: the web front, which serves the page, the viewer, /api/v1/… and the results under /results. When the node is not routable from your workstation, forward it, either through a host that reaches the node:

ssh -L <web-port>:<node-address>:<web-nodeport> <ssh-host>

or with kubectl:

kubectl -n <namespace> port-forward svc/htrflow-web <web-port>:8081

Then http://<workstation>:<web-port>/ is the campaign browser, and resultsUrl must be http://<workstation>:<web-port>/results, the address the browser uses. <ssh-host> is any machine you can reach that reaches the node; the -L target resolves on its side. Exposing the web front without a tunnel is covered in View results.

Resources that already exist on the cluster

A model-cache PVC, nvidia RuntimeClass or device-plugin DaemonSet applied by hand must be adopted into the release or left outside it (modelCache.create, and nvidiaDevicePlugin.enabled in the devstack chart; Deploy → Options; the commands are in each chart's README). A cache PVC root-running pods wrote to needs a one-time chown -R 1000:1000 before the first warm-up.

Upgrading the live release

helm upgrade <release> charts/htrflow-batch -n <namespace> --reset-then-reuse-values \
  --set web.image=<registry>/htrflow-web@sha256:<digest>
make psa-labels

The same --reset-then-reuse-values rule holds for charts/htrflow-devstack (Deploy → Upgrading).

Gotchas

• A node shared with other services may need host-level settings. Raise the inotify instance limit — when other services exhaust it, the kubelet fails to register the node without a clear error — and prefer absolute over percentage-based kubelet eviction thresholds on a large shared disk. The values belong to the host's configuration, not this repository.
• helm template needs network.apiServer.cidr, and wants network.nodeCidrs: the NetworkPolicies take both from lookup calls that return nothing without a cluster. The render fails without the API server CIDR, and without node CIDRs its public-egress rule excludes no node addresses.
• A pipeline's image must contain htrflow_batch.warmup. Every campaign pod's init container waits for the marker file the warm-up writes, so a pipeline pinned to an image without that module never starts. Pipelines are immutable: new work needs a new pipeline id on a current image.
• make install-devstack NVIDIA_DEVICE_PLUGIN=false refuses while GPU pods run. Disabling the plugin deletes the chart-managed nvidia RuntimeClass and device-plugin DaemonSet, which takes down any GPU pod running on them. The target looks for a pod outside kube-system that is Running or Pending and uses runtimeClassName: nvidia or requests nvidia.com/gpu, and exits non-zero if it finds one, or if it cannot list the pods to find out; FORCE=1 skips the check.
• RustFS is single-disk. The results bucket is one PVC on the node (rustfs-data, sized by rustfs.storage.size, kept on uninstall). Fine for iteration; not an archive.