Troubleshooting

Symptom first, then what to run, then the fix. <namespace> is the release namespace, <campaign> a campaign Job's name, <results-url> the release's resultsUrl and <web-front> the address in front of it. Why each mechanism behaves as it does is in How it Works.

Quick lookups

Result files answer 401 without a login. Log in once with a store account, keeping the session cookie in a file (Origin must be the site's own address, or the login is 403):

curl -s -c jar -H "Origin: <web-front>" -H "Content-Type: application/json" \
  -d '{"username": "<user>", "password": "<password>"}' <web-front>/results/_login

The Quickstart stack

A campaign reads "Queued" and nothing is admitted

"Queued" is not a Kueue state: the campaign browser shows it for a Job that is suspended and has no finished volume yet. That covers a campaign waiting its turn, one paused in git before its first volume finished, and one that can never be admitted.

Check

kubectl -n <namespace> get workloads                 # ADMITTED empty = still waiting
kubectl -n <namespace> describe workload <workload>
kubectl get clusterqueue <queue>-cq -o yaml          # pendingWorkloads, flavorsUsage
kubectl -n <namespace> get events --sort-by=.lastTimestamp
kubectl -n kueue-system get pods                     # is Kueue itself running?

To find a Job's Workload:

JOB_UID=$(kubectl -n <namespace> get job <campaign> -o jsonpath='{.metadata.uid}')
kubectl -n <namespace> get workloads -l "kueue.x-k8s.io/job-uid=$JOB_UID"

Fix, by cause:

Pods stuck in Init:0/1, or volumes failing on a missing marker

Every campaign pod waits in its warmup-wait init container for its pipeline's warm-up marker, for at most warmup_wait_seconds (default 900). Past that it exits 13 and the index fails without retry.

Check, in this order:

# 1. The index that gave up names the marker it waited for.
kubectl -n <namespace> get pods -l batch.kubernetes.io/job-name=<campaign> \
  -L batch.kubernetes.io/job-completion-index
kubectl -n <namespace> logs <pod> -c warmup-wait
# no warm-up marker at /data/warmup/<pipeline>.done after 900s: …

# 2. The warm-up Job says why it never wrote one.
kubectl -n <namespace> get job htr-warmup-<pipeline>
kubectl -n <namespace> logs job/htr-warmup-<pipeline> --tail=50
kubectl -n <namespace> get pods -l batch.kubernetes.io/job-name=htr-warmup-<pipeline> \
  -o jsonpath='{.items[*].status.containerStatuses[*].state.terminated.message}'

# 3. What is on disk, from a running pod of the same pipeline.
kubectl -n <namespace> exec <running-pod> -- ls -l /data/warmup

The warm-up's message also shows on the campaign card's warm-up chip.

Fix, by what the warm-up Job shows:

A Complete warm-up with no marker: first check that the warm-up Job has the same runtimeClassName, nodeSelector and tolerations as the campaign Job. With two or more GPU nodes, a ReadWriteOnce cache can be filled on one node and read on another; use a ReadWriteMany class (The Wrapper → The model cache). Once fixed, delete the Complete Job by hand, since an unchanged apply leaves a Complete warm-up alone:

kubectl -n <namespace> delete job htr-warmup-<pipeline>

then re-run the apply. Volumes whose index already failed on the missing marker do not run again in that campaign; see Re-run a failed volume.

A volume failed

The campaign card gives one sentence per failed volume, and its run log is one click away on the same row.

A volume whose state is failed has spent its retries, and the card says so; nothing will run it again in that campaign.

Re-run a failed volume

1. Fix the cause: a model, a manifest URL, a pipeline bug.
2. Add the volume to a new campaign file. The old campaign cannot be edited: campaigns are append-only, and removing the volume from it is refused too.

On the same pipeline id, the apply holds the new campaign back while a campaign that shares the volume is still running, since both would write the same results; it says so and starts once the old one finishes. Under a new pipeline id it starts at once.

Cannot log in

The login page words the results proxy's answer. Each has one cause:

Check

kubectl -n <namespace> get deploy htrflow-results
kubectl -n <namespace> logs deploy/htrflow-results

Fix, by what the page says:

The campaign browser shows no progress for a running volume

The read API reads each running volume's progress.json through the results proxy, with your session. A volume shows nothing when the proxy cannot reach the store or your account may not read that volume.

Check: open the volume's progress.json under /results/ in the same browser session. A 403 means the store refuses your account; a 502 or 504 means the proxy cannot reach the store (check network.s3Cidrs and the S3 Secret's S3_ENDPOINT, and S3_VERIFY_TLS for a certificate the proxy does not accept); a 401 means the session ended, so log in again.

Fix: whichever the answer names. Nothing about resultsUrl has to resolve from inside the cluster.

Logs or ALTO views are refused

The run viewer and /alto only read addresses under the chart's resultsUrl. Runs published under a different base (a changed chart value, or a converter.yaml results_url that does not match) are refused. Make the two equal (View results).

The apply exits non-zero

1 is a pause that is not enforced, nothing applied, or a campaign record that could not be read; 3 is some objects refused with every pause holding. The message on stderr names each object. What each refusal means and what to do is in htrflow-campaigns CLI.

The model cache

There is no API for the cache, only the filesystem. A campaign pod mounts only its own recipe's directory, read-only. To see or change the whole cache, start a debug pod that mounts the PVC (on a ReadWriteOnce volume it must land on the node that holds it; pin it with nodeName if needed):

kubectl run htr-cache-debug -n <namespace> --rm -it --restart=Never --image=busybox \
  --overrides='{"spec":{"containers":[{"name":"debug","image":"busybox","command":["sh"],"stdin":true,"tty":true,"volumeMounts":[{"name":"data","mountPath":"/data"}]}],"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"<cache-pvc>"}}]}}'

Each recipe is a directory /data/<pipeline-id>-<recipe sha256>/. A pipeline's current recipe hash:

kubectl -n <namespace> get job htr-warmup-<pipeline-id> \
  -o jsonpath='{.spec.template.metadata.annotations.htrflow\.riksarkivet\.se/recipe-sha256}'

After replacing the cache PVC, delete every htr-warmup-* Job and apply, so they warm the new volume. A model missing from the cache during a run fails the attempt as transient; retries succeed once the cache is fixed.