htrflow-campaigns CLI

htrflow-campaigns is the converter's command line. It reads a campaigns repo, checks it, renders it to Kubernetes objects and applies them. The file formats it reads are in Campaign & Pipeline YAML. What a render produces, field by field, is in Rendered objects.

Source: packages/converter/src/htrflow_converter/cli.py.

Commands

make campaigns-apply DIR=<repo> runs uv run htrflow-campaigns apply <repo> --out <repo>/rendered --namespace <namespace>, the namespace being HTR_NAMESPACE. PRUNE=1 adds --prune, and ALLOW_EMPTY=1 adds --allow-empty.

apply talks to the API server through the official Kubernetes client, with server-side apply and no kubectl. It authenticates from $KUBECONFIG or, in a pod, from the mounted ServiceAccount token. The chart renders a suitable ServiceAccount behind apply.rbac.enabled (Chart Values).

export: publishing finished volumes

htrflow-campaigns export --pipeline <id> --namespace <ns>
    (--volumes ID,ID | --volumes-file FILE) --to <dir>
    [--results-url URL] [--min-quality 0.75 | --no-quality-filter]
    [--empty-pages 1] [--manifests DIR] [--fix-positions] [--user NAME]
    [--report FILE] [--dry-run]

What it reads. Each volume's results at <results-url>/<namespace>/<pipeline>/<volume id>/, the same place the viewer reads them. No campaigns repo is needed: the pipeline is the one the campaign file names, and the volumes are those it lists. --volumes takes ids separated by commas and --volumes-file one id per line, with blank lines and # comments skipped (a UTF-8 byte-order mark too); both may be given, and the flag's ids come first. An id listed twice (letter case ignored), or one that is not a volume id, refuses the run.

What it writes. For each volume, a folder <dir>/<volume id>/ holding one ALTO file per image, named <volume id>_<nnnnn>.xml after the page's own source image id, never after its place in the canvas order. Inside the file that id is the fileName, and the Page's ID is its five-digit image number behind an underscore (_00004), whatever name the run gave the page. A volume is published whole or not at all: it is built beside the target and renamed into place only when every page is there, so the real folder name never holds a partial volume, and an earlier export of the same volume is replaced as a whole. Only a whole image series is published: a volume made from an images: list, or whose image numbers are not exactly 1 to its page count, fails. So does a volume whose results change during the export: its manifest.json differs at the end from what was read first, or a page recorded as transcribed has no ALTO. Point --to at a folder inside the share, not at the bare mount point: an unmounted share leaves the mount point as an empty local directory, while a folder inside it is then missing and the run is refused. The files and folders are written with the person's own umask and ownership; the publishing side must be able to read and remove them, so check the share's setup (group, ACLs, squash) before the first export.

The quality rule. A volume is exported only if the mean predicted quality of its pages reaches --min-quality (default 0.75). The mean counts pages that have a score other than exactly 0 and are not among the empty pages. A volume with no such page is skipped as no scores; --no-quality-filter exports volumes whatever they score.

The empty-page rule. --empty-pages (default 1, the first page) names pages, counted from 1 and written as 1,3-4, that are published as an empty ALTO whatever was read; none empties nothing, and positions past a volume's end are ignored. A page with no ALTO (a page that failed) is published empty too, so every volume holds exactly its manifest's pages. An empty ALTO states only the page's size and its source file name.

Scaling. Results are read at the size the pipeline worked at, and the export rewrites every coordinate to the scan's full resolution, read from the image server. A volume whose page proportions differ from its scan's by more than one per cent fails instead of being published stretched.

Positions on the page. htrflow writes two positions from the wrong corner: a word's box is counted from the corner of its text block instead of the page's, and a block's outline is moved by that corner once more. Word boxes come only from a pipeline that recognises word by word; blocks' and lines' own boxes are right. Without --fix-positions both are published as htrflow wrote them, so a search hit is drawn up and to the left of its word. With it, the export puts both where they are on the page before it scales. It tells the two forms apart by the file's own geometry: an outline whose corner is exactly twice its block's is moved back, and a page's words are moved by their blocks' corners only when that puts more of them inside their lines. A file that is already right is published as it is, so the option is safe to pass for every volume. The results themselves are not changed.

Without the image server. The full resolution is not in the results, so by default each page's size is read from the image server, one request per page. Where the export runs without access to the image server, download each volume's IIIF manifest elsewhere, save it as <volume id>.json in one folder, and pass that folder as --manifests. Each page's size is then its canvas's in that file, and the image server is never asked. A volume fails, with nothing written, when its file is missing or unreadable, when the file's own id is not the manifest URL the volume's results recorded, when a page's image is not on exactly one canvas, when a canvas has no size, or when an image states a size other than its canvas's.

The login. The export reads only through the results proxy at --results-url (else HTRFLOW_RESULTS_URL), the site's address ending in /results, for example https://<web front host>/results. It logs in once with --user or HTRFLOW_USER, else a prompt, and the password from HTRFLOW_PASSWORD or a prompt without echo, never an argument. The results address must be https, except plain http to the workstation's own loopback address, so the password never crosses the network in clear. A volume the account may not read is skipped as forbidden; an expired session is renewed once, and then the run stops.

Progress. While it runs, the export writes one line per volume to stderr as each one is done, with its outcome, page count, time taken and, for a failed volume, the reason, then a last line with the total time. In a terminal, a counter for the slow steps (reading page sizes, building the ALTO) is redrawn in place; written to a file or a pipe, only the finished lines appear. The report below goes to stdout, so 2>/dev/null hides the progress and leaves the report.

The report. One row per volume listed is printed as a table and written as CSV to --report (default ./export-<pipeline>.csv in the current directory, never inside the target) with the columns volume, outcome, pages, pages_with_text, pages_empty_filled, pages_empty_chosen, quality_mean, quality_pages and reason. The outcomes are exported, below threshold, not finished, no scores, forbidden and failed. When every volume reads as not finished, a note says to check that --namespace, --pipeline and --results-url point at the deployment that ran those volumes. --dry-run reads and checks everything, writes nothing to the target and still writes the report; a volume that would be exported is reported as would export.

Exit status. 2 when the run is refused before any request: the target is not an existing directory (it is never created, so an unmounted share cannot become a local folder), the report's folder or the --manifests folder does not exist, no volume is listed or the list cannot be read, an id is listed twice or is not a volume id, the namespace or pipeline is not a valid name, no results address is given or it is not https or does not end in /results, no user name or password could be read, or an option is out of range. 1 when any volume failed or was forbidden, or the run stopped (the session was lost or the target became unusable); the report then still lists every volume, the unfinished ones as failed. Otherwise 0: every volume was exported (or would export), below threshold, not finished or had no scores.

The design is in the campaign export spec among the design specs.

retry: starting failed volumes again

htrflow-campaigns retry <campaign> --namespace <ns>
    [--pipeline ID] [--include-permanent] [--no-old-apply]
    [--results-url URL] [--user NAME] [--dry-run]

What it reads. The campaign's record in the cluster (its volume list and its Job's settings) and, from the results proxy, whether each volume has a manifest.json. A volume without one is unfinished; its run log says whether it failed for good (permanent), will be tried again (retry), or never started. Never-started volumes are retried. Volumes that failed for good are skipped, and named in the retry's record, unless --include-permanent takes them.

Credentials are the ones export takes: --user, else HTRFLOW_USER, else a prompt; the password from HTRFLOW_PASSWORD or a prompt, never an argument. The results address is --results-url, else the RESULTS_URL on the original Job.

What it starts. A new campaign, <campaign>-retry-<n>, with one more n than the highest retry of that campaign. It is the original's Job with only its name, labels, volumes and counts changed (and suspend dropped, so a paused original does not leave its retry paused), so its settings are copied: a cause that lives in the settings, such as a too-small workdir, is not fixed by a retry. A retry of a retry is named after the original, and one of a split campaign after its first part (<name>-part1-retry-<n>). Its ConfigMap records the volumes (volumes.txt), the ones left out (the annotations <label-domain>/retry-skipped, capped at 64 KiB, and retry-skipped-count, on the first part's ConfigMap only), who ran it and when. A large retry is split into parts the way render splits a campaign. If the cluster refuses a write part-way, the ConfigMap just written for that part is deleted, and the error names every object already created and still live.

campaign kyrk: 4 volumes, 1 done, 3 not done
  2  retry      no run log: the volume never started
  1  permanent  manifest: canvas 4 has no image (skipped; --include-permanent takes it)
would start: Job/kyrk-retry-1 with 2 volumes

The summary is the tally, then one line per kind of failure, largest first. --dry-run prints the would start: line and creates nothing; a real run prints started: Job/kyrk-retry-1 with 2 volumes instead. With nothing to start, the summary is followed by nothing to retry: all N volumes of <campaign> are done, or nothing to retry: every volume not done failed for good (--include-permanent takes them), and the exit code is 0.

It refuses, with one sentence on standard error and nothing created, when:

• the campaign, or an earlier retry of it, is still running;
• the pipeline's warm-up Job is missing or not complete, or no longer runs the image the campaign ran (a retry runs only what the repo approved: run the volumes with --pipeline on the pipeline that replaced it, or as a new campaign file);
• --pipeline names a pipeline whose ConfigMap or warm-up Job is not in the namespace, or, with no campaign on it to copy, one that runs at another size than the campaign's pipeline (or the campaign's pipeline is gone, so its size is not known);
• another running campaign on the pipeline holds one of the volumes, or has no volume list to check;
• there is no campaign of that name in the namespace;
• the campaign has no stored Job to copy, its own Job is gone and no other campaign on the same pipeline and image has one either, or the stored Job cannot be read;
• the cluster's last apply ran a converter whose prune does not know retries, or, on a pipeline pipeline add added, does not keep added pipelines, unless --no-old-apply is given.

A campaign applied by a converter older than this feature has no stored Job. While its own Job is there, that is copied. Once the TTL has reaped it, the settings are copied from another campaign's Job, live or stored, on the same pipeline and image, the newest first: everything in a campaign Job but its name, counts, volume list, window and priority comes from the pipeline and converter.yaml, and the first three are rewritten anyway. The window is that campaign's; the priority is dropped, so the retry runs at the default class. The summary says so: settings: copied from campaign <other>'s Job, on the same pipeline and image (...); default priority.

Upgrade the Argo CD hook's converter first and let it apply once: until then retry refuses, because that apply would prune the retry. Where no hook runs and every apply is made by hand, --no-old-apply starts the retry anyway, on the operator's word that no apply by an older converter runs until it is done; it says that on standard error and claims no mark on the Lease. An apply by this version stores the Job on every finished campaign that lacks one and whose campaign file is still in the repo: it renders that Job anyway, and adds it to the campaign's ConfigMap without running anything again. The stored Job is what the repo renders at that apply, so a setting changed since the campaign ran, such as a raised workdir, is the one a retry gets. A stored Job that is already there is never replaced.

On another pipeline. --pipeline <id> runs the volumes on a pipeline the repo has applied in the namespace, or pipeline add has added there, instead of the one the campaign ran on. Done means done there: a volume counts as done when that pipeline's results hold its manifest.json, so the first such retry takes every volume and a later one picks up where it stopped. Results are kept per pipeline, so the old pipeline's stay where they are. The settings come from a campaign that has run on that pipeline and its image, live or stored, the newest first, with the campaign's own priority; with none, the campaign's own Job is moved onto the pipeline from its warm-up Job: the image, the pipeline ConfigMap, the model-cache directory and the warm-up marker. A pipeline file's own max_seconds and ttl_seconds_after_finished are not on the cluster, so a moved Job keeps the deadline and TTL it had. The summary says which:

pipeline: demo-v1 -> demo-v2, image <image@digest>
settings: kyrk's Job moved onto demo-v2; its per-volume deadline and TTL stay as they were, ...

The retry is recorded on the new pipeline, so retrying it again stays there. A pipeline the repo applied has to keep its file in the repo while the retry runs, as for any campaign.

Permissions. The operator running retry needs: list and get on Jobs and ConfigMaps; get, update and create on the Lease htrflow-campaigns-apply; create on Jobs and ConfigMaps (dry runs included); delete on ConfigMaps, for taking back the one written for a part whose Job was refused; and read on the results prefix.

Prune keeps a retry for as long as the original campaign's file is in git, and removes both when that file leaves git.

pipeline add: a pipeline with no repo

htrflow-campaigns pipeline add <file> --namespace <ns> --like <id>
    [--no-old-apply] [--dry-run]

Puts one pipeline file in the namespace, so that retry --pipeline can run a campaign's volumes on it, with no campaigns repo and no commit. The file is the shape of pipelines/<id>.yaml, and its id is the file name without .yaml.

What it makes. The two objects apply makes for a pipeline: its ConfigMap and its warm-up Job, which downloads the models into the model cache. The file is validated as validate validates one in a repo. What the two objects take from converter.yaml (the data volume, the GPU RuntimeClass, the node placement, the Hub token's Secret) is the namespace's, and on every warm-up Job apply made, so the warm-up of the pipeline --like names is copied, with the new pipeline's id, image, recipe and model-cache directory set. The size is --like's too: what a size gives a pod is in converter.yaml, which is not read here. Both objects carry the <label-domain>/added label.

pipeline demo-v2: image <image@digest>
settings: copied from pipeline demo-v1's warm-up Job (data volume, GPU RuntimeClass, node placement, Hub token Secret), and its size
added: ConfigMap/htr-pipeline-demo-v2
added: Job/htr-warmup-demo-v2
next: once Job/htr-warmup-demo-v2 has completed, htrflow-campaigns retry CAMPAIGN --namespace <ns> --pipeline demo-v2

--dry-run prints would add: lines and creates nothing. Adding a file whose pipeline is there already with the same recipe changes nothing.

It refuses, with nothing created, when:

• the file does not validate (one line per problem, as validate prints);
• --like names a pipeline whose ConfigMap or warm-up Job is not in the namespace, or was not made by htrflow-campaigns;
• the file names a size: other than --like's;
• a pipeline of that id is in the namespace with another recipe, or only one of its two objects is;
• the cluster's last apply ran a converter whose prune does not keep added pipelines, unless --no-old-apply is given.

Prune keeps an added pipeline. No file renders it, so an apply's prune would delete it, and with its ConfigMap the retries on it could not start their pods. A prune by a converter that has pipeline add keeps every object with the added label, and every apply by one marks the Lease htrflow-campaigns-apply to say so. pipeline add, and retry on an added pipeline, refuse until that mark is there: upgrade the Argo CD hook's converter and let it apply once. --no-old-apply adds the pipeline anyway, on the operator's word that no apply by an older converter runs until the retries on it are done.

Into the repo later. An apply of a repo that holds the same file takes the two objects over, and its render has no added label, so from then on the pipeline is the repo's like any other. A pipeline id names one recipe for good, so the file must be the same; an apply refuses another recipe under that id while a Job runs on it.

Permissions. As for retry: get on ConfigMaps and Jobs; get, update and create on the Lease; create on ConfigMaps and Jobs (dry runs included); and delete on ConfigMaps, for taking back the ConfigMap when the API server refuses the warm-up Job.

What apply does, in order

1. Render the repo. A render that does not pass stops here.
2. Check the namespace against --namespace, and refuse an empty render under --prune without --allow-empty.
3. Take the Lease htrflow-campaigns-apply for the whole run (One apply at a time).
4. Check against the live cluster, before anything is sent. Any of these stops the whole apply with exit 1:
   • a campaign whose live ConfigMap has a different volume list, pipeline or image
   • a pipeline whose live steps differ while a campaign Job that has not ended still mounts it
   • a live, unpaused campaign Job whose pod count the render would change (Changing a window)
5. Record check. For each campaign, read the live Job and write the ending it shows into the campaign's status record. A campaign that is finished and whose volume list has not moved is left alone (Finished campaigns).
6. Volume check. Hold back a campaign that shares a volume with another campaign on the same pipeline that is still running, or starting in this apply.
7. Dry-run each campaign Job (dryRun=All, admission webhooks included). A campaign whose Job would be refused keeps its ConfigMap unchanged too.
8. Apply pipelines: each pipeline's ConfigMap and warm-up Job. They go first because a campaign Job mounts its pipeline's ConfigMap.
9. Apply campaigns: each campaign's ConfigMap, then its Job.
10. Pause sync: set each campaign's Kueue Workload spec.active from git (Pausing). This reaches campaign Jobs the API server refused too.
11. Prune, with --prune only.

The pause sync runs before the prune on purpose. A missing pause burns GPU time now, so a prune problem must never stop it.

Every action prints one line: applied: <Kind>/<name>, replaced: …, pruned: …, or a sentence on stderr for a refusal.

rendered/

rendered/
  pipelines/<id>.yaml     # ConfigMap htr-pipeline-<id> + Job htr-warmup-<id>
  campaigns/<name>.yaml   # ConfigMap campaign-<name> + the campaign's Indexed Job
  sync.yaml               # ConfigMap htrflow-campaigns-render: the render's digest

The campaigns repo's CI renders this on main and commits it. Never edit it by hand.

• Only htrflow-campaigns apply applies it. Never kubectl apply it, and never let Argo CD apply it. A finished campaign's Job is deleted at its TTL while its file stays in rendered/, and a tool that makes the cluster match the directory creates that Job again and runs every volume again. It would also skip every check above.
• Every object is an Argo CD Skip hook. Each object under pipelines/ and campaigns/ carries argocd.argoproj.io/hook: Skip. An Application that syncs rendered/ applies, heals and prunes none of them.
• The committed rendered/ is the record. --out says only where a render is written. validate and render always hold the repo against its own committed rendered/, so rendering into a fresh directory does not make a running campaign new.
• render removes files it did not produce. Deleting campaigns/<name>.yaml deletes rendered/campaigns/<name>.yaml too. render writes the whole render beside --out and moves it in only when it is complete. Anything in --out other than pipelines/, campaigns/ and sync.yaml is left alone.
• The static shape is in the skeletons. The four objects are real YAML under manifests/, with placeholders that render.py fills in.

With Argo CD

Argo CD syncs one object and runs the command as a hook:

• The Application's source is the campaigns repo, with directory.recurse: true, directory.include set to {rendered/sync.yaml,argocd/*.yaml}, and automated sync on.
• rendered/sync.yaml is a ConfigMap holding a digest of the render. A render that changes anything changes it, so the Application goes OutOfSync and syncs. Without it, an Application whose other objects are all Skip hooks is never OutOfSync. It carries no converter label, so a prune leaves it alone.
• The hook is a PostSync Job that runs htrflow-campaigns apply --prune on a checkout of the campaigns repo (The hook manifest).

A refresh, a self-heal or a re-sync with no new render changes nothing. Cancelling is the command's --prune: Argo CD never applied a campaign object, so it has none to prune.

The hook manifest

htrflow-campaigns init writes it as argocd/apply.yaml (in the template). It is a PostSync Job on the htrflow-campaigns image with three steps:

1. An init container clones the campaigns repo over HTTPS with dulwich (pure-Python git; the image has no git binary and no shell).
2. A second init container runs htrflow-campaigns validate --rendered /repo.
3. The main container runs htrflow-campaigns apply --prune --namespace <the hook's namespace> /repo.

What to set:

Why validate --rendered. The hook clones the tracked branch, not the revision Argo CD synced, because a hook Job cannot reliably learn that revision. So the check renders the checkout and refuses it unless the result matches the committed rendered/, compared as parsed objects. A commit merged after CI's last render fails the hook and applies nothing. CI's render commit for it then changes sync.yaml and starts the next sync. The check also fails when CI's CONVERTER_REF and the hook's image render different objects, so keep them on the same converter release.

Exit codes in a hook. The hook fails the sync on any non-zero exit. Exit 3 still changed the cluster, and its summary line names what is left to fix. Read the summary line rather than the code (Exit codes).

Pausing

suspend: true in a campaign file renders spec.suspend: true on its Job. But Kueue owns spec.suspend on a Job it has admitted and undoes any edit within seconds. So the apply enforces the pause on the Workload instead (Queueing → Pause):

• Every campaign's Workload gets spec.active: false for a suspended campaign, true otherwise. The patch is idempotent.
• Deactivating evicts the campaign's pods and keeps every finished index. kubectl get job then shows suspend: true.
• Reactivating continues at the next unfinished index.
• active: true is written for every unpaused campaign. A Workload Kueue deactivated on its own (a requeue limit, maximumExecutionTimeSeconds) is re-admitted at the next apply. Nothing stays paused unless git says so.
• A new paused campaign has no Workload yet when its Job is created. The apply waits up to --pause-wait seconds for it, and exits 1 if it never appears: re-run the apply. An unpaused campaign whose Workload is missing is skipped, since a Workload that does not exist is not admitted.
• Resuming a campaign Kueue never admitted keeps its Job suspended through the apply, under a second field manager (htrflow-campaigns-suspend), until Kueue admits it.
• A campaign Job the API server refused still has its pause synced. A converter upgrade can make every live campaign Job refused, and a paused campaign must still stop.

With Argo CD, a merged suspend: true takes effect on the sync its render triggers.

Prune: cancelling a campaign

Deleting a campaign file removes its rendered file, but the objects stay in the cluster until an apply prunes them:

• apply --prune lists every Job and ConfigMap in the namespace that carries the converter's managed-by=converter label and deletes the ones this render did not produce. Every object the converter renders carries that label. So does the status record, which a prune removes together with its campaign.
• It is opt-in. Without --prune, a deleted campaign's Job simply stays. The Argo CD hook always passes it.
• Never prune from a partial checkout. Anything missing from the render is deleted.
• An empty render is refused. A render with no campaigns at all under --prune looks like an empty campaigns/, a mistyped directory or a checkout that never happened. Pass --allow-empty when retiring the last campaign is what you mean. --dry-run still shows what such a prune would delete.
• A deleted pipeline file takes its ConfigMap and warm-up Job with it.
• A pipeline pipeline add made is kept: no file renders it. See pipeline add.

Results in S3 are never touched by a prune, or by anything else here.

One apply at a time

Two applies at once, such as the Argo CD hook and a hand-run apply, would interleave. A prune from the older checkout would delete what the newer one just created. So:

• apply holds the coordination.k8s.io Lease htrflow-campaigns-apply in the namespace for its whole run, and releases it however it ends, SIGTERM included.
• A second apply while the Lease is held sends nothing, names the holder and exits 1. Re-run it once the first is done.
• The holder renews the Lease as it goes. A Lease not renewed for ten minutes belongs to an apply that died, and the next apply takes it over.
• An apply that cannot renew, or finds the Lease taken over, stops and exits 1, since another apply may be running.
• Both the renewal times and the takeover judgement use the API server's clock, never the local machine's.

Every request has a 5 s connect and a 60 s read timeout, so a half-closed connection cannot hang an apply for ever. A 429 or a 5xx is retried three times, after 1, 2 and 4 s, since a control plane being upgraded answers with those; any other refusal is an answer about the request and is not retried.

--namespace guards the other side: the chart's policies match the release namespace only, so an apply from a kubeconfig with wider rights must not put Jobs anywhere else. The Argo CD hook passes its own namespace.

Checks apply makes against the cluster

validate and render hold a repo against its committed rendered/. That can be missing or behind the cluster, so apply checks the live objects as well.

Append-only and immutable recipes

The rules themselves are in Campaign & Pipeline YAML → Immutability.

Changing a running campaign's window

Kueue compares an admitted Job's pod count (the smaller of parallelism and the volume count) with its Workload. When they differ, it stops every running pod and queues the campaign again. So:

• validate and render print a warning: when a window change would change a rendered, unpaused campaign's pod count, and go on.
• apply refuses the change for a live, unpaused campaign Job: campaign <name> runs <n> pods at a time and would now run <m> … nothing was applied, exit 1.

The safe way: pause the campaign, change the window once the pause is applied, then resume it. A paused campaign's Workload is updated in place.

A size's flavor against the cluster's

A campaign whose size names a flavor is sent only while converter.yaml's flavors are the ClusterQueue's, names and node labels alike. apply reads the LocalQueue converter.yaml names, its ClusterQueue and each of that queue's ResourceFlavors, by name. When they differ, the campaign is left as it was with a sentence naming each difference (a flavor one side has and the other does not, a label value spelt otherwise), and the apply exits 3. Campaigns with no flavor, and the pipelines, go out as usual. An identity that may not make those reads (a kubeconfig without the rights the chart's apply identity has) gets a warning, and the campaign goes out unchecked. Several sorts of GPU says why a difference matters.

Two campaigns on one volume

Results are keyed by pipeline and volume, so two campaigns on one pipeline must not run the same volume at once. apply compares each campaign's volume ids with those of every other campaign on the same pipeline whose Job has not ended, and with campaigns earlier in the same apply. A campaign that shares one is left as it was, named with the other campaign and the volumes, and the apply exits 3. Once the other campaign has ended, the next apply sends it. If the running campaigns cannot be read, each new campaign is held back the same way.

Listing a failed volume again in a new campaign is how it runs again, and that is fine once the old campaign is over.

Finished campaigns are not run again

apply reads each campaign's live Job and its status record (campaign-<name>-status) before it sends anything:

• A live Job decides. While the Job exists, it alone says whether the campaign is over, and its ending is written into the record.
• With no Job, the record decides. A campaign the record says is Succeeded, Failed or PartiallyFailed, whose volumes.txt has not moved, is left alone: campaign kyrk finished <date>, unchanged, left alone (120/120 volumes).
• A record counts only if it names this campaign's Job. apply stamps the uid of the Job it created on the campaign ConfigMap (job-uid). A record about another Job, or about a Job never created, is not believed: apply says so and applies the campaign. A ConfigMap from before the stamp existed carries no uid, and its record is believed.
• Left alone is not left unrecorded. A finished campaign's ConfigMap that lacks the stored Job (job-spec) a retry copies, or a job-uid while its Job is still there, gets the missing one and nothing else: its volumes, its apply time and its Job stay as they were.
• A Job or record that cannot be read leaves that campaign exactly as it is, named, with exit 3.
• There is no --force. A campaign that should run again is a new campaign file.

What the record holds is in Campaigns → The record.

When the API server refuses an object

apply sends each object on its own. One refusal is one object's problem: it is named on stderr, everything else is still applied, and a summary line lists what was left unchanged. A campaign's ConfigMap and Job are one change: if the Job's dry run is refused, the ConfigMap is kept as it was too.

An object that already exists without the converter's managed-by: converter label (made by hand, or by something else) is never overwritten, whoever runs the apply: it is refused as … exists … and was not made by htrflow-campaigns, and so are the campaign pair and the Jobs that would mount it. Rename the pipeline or campaign, or remove that object.

Job htr-warmup-demo: the pod template changed and a Job's pod template is immutable once the Job exists — a pipeline id is a permanent name for a recipe, so a changed recipe is a new pipeline file, and a Job that has to change is deleted and created again
2 of 6 objects were refused by the API server and are unchanged: ConfigMap/campaign-kyrk, Job/kyrk — the other 4 were applied (exit 3)

A pause sync that did not reach a Workload (the patch was refused, or a refused Job's live Job could not be read) has its own closing line: the pause sync did not reach the Kueue Workload of Job/kyrk; see above (exit 1). The code is 1 when git says that campaign is paused, since the pause is not enforced, and 3 when it only had to stay running (unless something else in the apply already makes it 1).

A Job's pod template is fixed once the Job exists. Two different changes move it: a recipe edit, and a converter upgrade or converter.yaml setting that reaches every Job (the RuntimeClass, the Hub token's env). They get different answers:

• A warm-up Job is replaced. It holds no state beyond its marker on the cache PVC, so apply deletes it, creates it again and prints replaced: Job/htr-warmup-<id> — its pod template changed ….
  • A warm-up that is running is left alone and reported, since deleting it would kill the download. If the change was to the recipe, re-run the apply once the old warm-up finishes, before the new recipe's campaigns run out of warmup_wait_seconds.
  • A warm-up that has failed is replaced even with an unchanged template, since a failed Job never runs again: replaced: Job/htr-warmup-<id> — it had failed ….
• A campaign Job is never replaced. Its completed indexes are the campaign, and deleting it would start every volume over. It is reported, left as it is, and the apply exits 3. Its pause still follows git.

Exit codes

The codes are a precedence, highest first: 1 beats 3 beats 0.

export has its own codes, set out above: 0 when no volume failed or was forbidden, 1 when one was or the run stopped, 2 when it was refused before any request.

retry has its own codes, set out above.

validate and render exit non-zero on any problem, with one line per problem (When something is wrong).