Cache layer

A proposal, not built: a read-through cache layer between campaign pods and the IIIF origin, beyond the S3 image cache that already exists. Build it only when measurements from real campaigns show it is needed. This page covers the evidence that would justify it, two ways to build it, what each one promises, how each one fails, and the spike that decides between them.

What happens today

The wrapper fetches pages straight from the IIIF origin (Page flow, The Wrapper):

• Downloads start as soon as the stream is built, so they overlap the model load.
• A bounded pool (DOWNLOAD_CONCURRENCY) fills a bounded lookahead window (LOOKAHEAD_PAGES) on a memory-backed workdir. The GPU consumes pages as they land, and each image is deleted once its outcome is recorded.
• Each image URL comes from the canvas's image service in the source manifest, capped at MAX_IMAGE_WIDTH. Volumes listed as plain images: are fetched at the URL given.
• An S3 image cache can be switched on (Deploy → Cache source images). Each page's image is looked for in a private bucket first and stored there after a download, so a retry or a re-run under a new pipeline reads the pages it holds from the bucket, not the origin (The Wrapper → Image cache). Without it, every such run fetches every page it processes again.

Streaming already caps the GPU's wait at about one page's download at the start, plus any moments the origin falls behind, and the S3 image cache already answers repeat fetches. The two variants below are the next step, for when the evidence shows idle GPUs or origin load: they add prefetching ahead of the queue, and shield the origin on a volume's first read too.

The evidence that would justify it

Every completed volume's manifest.json records the numbers needed (S3 Layout):

Fetch time itself is not recorded. Downloads run concurrently and ahead of the GPU, so their sum would not measure what they cost; stall does. Per-page download latency, if it is wanted, is a small addition to results.

Nothing aggregates these yet. A one-off over the manifests of one pipeline does it (use <namespace>/ in front only when the release sets S3_PREFIX):

aws s3 cp --recursive "s3://<bucket>/<namespace>/<pipeline>/" ./manifests \
  --exclude '*' --include '*/manifest.json'
find ./manifests -name manifest.json -print0 | xargs -0 jq -s '
  { volumes: length,
    wall:  (map(.wall_seconds) | add),
    stall: (map(.gpu_stall_seconds) | add),
    gpu:   (map([.results[] | select(.status == "ok") | .seconds] | add // 0) | add),
    bytes: (map(.bytes_fetched) | add) }
  | .stall_fraction = (.stall / .wall)'

Build the cache when one of these holds across an archive-scale campaign, not a handful of volumes:

• GPU idle. The aggregate stall_fraction stays above roughly ten per cent.
• Origin load. The origin rate-limits or fails under backfill: 429 and 5xx fetch errors in the run logs, and page failures they explain.
• Repeat fetches. The same volumes are fetched again and again, through retries, re-runs under new pipeline ids or several tenants. bytes_fetched summed per volume across runs shows how much. The S3 image cache is the answer built for this one; its hits and misses are in each manifest.json's image_cache.

What both variants keep

• Read-through. A page never prefetched is fetched on first read. The cache accelerates and is never a correctness dependency: if it is cold or gone, Jobs are slower, not broken.
• Width in the key. The requested width stays part of the cache key, so a cached image can never be served at the wrong resolution. The built S3 image cache deliberately does not do this: operators fixed its key as {ref}/{ref}_{page:05d}.jpg, with no width, so a cached image is reused at whatever width first stored it.
• Queue-aware warming. Warming at submit time would push hundreds of volumes through the cache hours before they run, and evict each other on the way. A small warmer instead prefetches only the next few volumes that will actually run:
• the pending indexes of admitted campaigns
• then the first lines of the queued campaigns, in admission order

It reads Jobs, Workloads and each campaign's volumes.txt ConfigMap, and drops warm data for completed volumes. Its death only makes Jobs colder.

Variant A: caching HTTP proxy

One proxy Deployment with its cache on a memory-backed emptyDir (or a disk, for reuse across campaigns), plus the warmer.

proxy_cache_path /cache levels=1:2 keys_zone=iiif:64m
                 max_size=<cache-size> inactive=24h use_temp_path=off;
server {
  listen 8080;
  location / {
    proxy_pass https://<iiif-origin>;
    proxy_ssl_server_name on;
    proxy_cache iiif;
    proxy_cache_valid 200 7d;
    proxy_cache_lock on;                          # collapses concurrent misses
    proxy_ignore_headers Cache-Control Expires;   # IIIF image URLs are immutable
    add_header X-Cache-Status $upstream_cache_status;
  }
}

Contract.

• The proxy serves GET <image-path> byte-identical to the origin. The full URL, including the width segment, is the cache key.
• The wrapper takes image URLs from the manifest, so there is no base URL to repoint. The wrapper needs a setting that maps <iiif-origin> to the proxy for image requests, and it falls back to the origin when the proxy errors.
• Campaign pods' IIIF egress can then narrow to the proxy.

Failure modes.

Variant B: Fluid, Alluxio and an index shim

A data layer that makes every volume a read-only directory on a FUSE mount, cached in memory on the GPU nodes. Campaign pods do no downloading.

What it rests on

• HTTP sources. Alluxio's web under-storage (WebUFS) mounts a plain HTTP(S) source, and Fluid's own samples mount a public HTTPS mirror as a Dataset this way.
• How WebUFS lists. It builds its namespace by parsing HTML directory index pages:
• a text/html page whose title carries a configured marker (such as Index of) is a directory
• configured parent-link names are skipped
• every remaining link is an entry

It is read-only: every write operation is refused. - File metadata. WebUFS takes a file's size from Content-Length and its mtime from Last-Modified on a HEAD, parsed with a configured date format. The timeout, date format, title markers and parent names are properties. - Warming. Fluid's DataLoad resource preloads a path of a Dataset into the runtime's cache. - The gap. IIIF serves JSON manifests and parameterised image URLs, not directory index pages, so it cannot be mounted directly. A small stateless index shim makes it look like one.

These come from Fluid's samples on accelerating web data access and data warm-up, and from the Alluxio web under-storage source.

Architecture

Kueue and Fluid do not conflict. Kueue decides when a Job runs. Fluid's webhook adds node-affinity preferences that decide where its pods land, near the cached blocks.

Warm path and miss path

Component contracts

Index shim. A stateless Deployment with two replicas behind a Service.

• Manifests are cached in memory (LRU, minutes), so the origin is hit once per listing or warm, not once per page.
• Bytes are proxied, not redirected. That keeps WebUFS on one host, applies the width cap and controls the headers.
• Configuration is the origin, the allowed widths and the manifest template. No disk and no state.

The metadata problem. Alluxio wants a size and an mtime at listing time. IIIF image servers derive images on demand and often send no Content-Length on HEAD. The options, in order of preference:

1. Stable fake sizes. The shim reports one constant size. The spike checks that Alluxio reads to the real end of the stream, without truncating or padding.
2. Size on first touch. The shim fetches each derivative once and remembers its real size. Per volume this is exactly the warm traffic, so the DataLoad pays for it, not the GPU.
3. Both fail. Variant B falls, and Variant A remains.

Fluid objects. One Dataset and one runtime serve the whole system:

apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
  name: iiif-volumes
  namespace: <namespace>
spec:
  mounts:
    - name: volumes
      mountPoint: web://iiif-shim.<namespace>.svc:8080/
  accessModes: ["ReadOnlyMany"]

• The AlluxioRuntime of the same name runs a worker per GPU node, with a memory tier on /dev/shm; a disk tier is what would make reuse across campaigns real. A short metadata sync interval lets new volumes appear without a remount.
• The warmer creates a DataLoad per volume it warms, targeting /volumes/<volume-ref>/w<width>/.
• The WebUFS mount options (timeouts, the Last-Modified format the shim must match) are pinned during the spike.

Campaign pod changes.

• The pod mounts the Dataset's PVC read-only, and the wrapper's input directory becomes volumes/<volume-ref>/w<width>/ on that mount.
• The fetch stage disappears: resume, run, verify, publish.
• Resume passes htrflow an inputs list, because pages cannot be deleted from a read-only mount.
• Outputs still go to the memory-backed workdir.

Memory

Images leave the pod: the lookahead window no longer lives on the workdir, so the pod's memory holds the resident models and page outputs only. Pod OOM risk stops scaling with image size. A huge volume churns the cache instead of killing a Job. The RAM moves to the Alluxio workers, which serve reads at memory speed on the same nodes.

Those workers are a standing Deployment, not queued workloads, so their memory is outside the Kueue quota. Size node capacity for both.

Failure modes

Open issues:

1. Content-Length behaviour. Unresolved until the spike runs.
2. FUSE in the GPU's critical path. A stalled read looks like a hung image read inside htrflow. The wrapper's guard watches htrflow's worker threads for death, not for a blocked read. Timeout ownership moves from the wrapper's own HTTP client to a FUSE daemon. Adopting Variant B needs a watchdog on output progress: the pod's activeDeadlineSeconds is a backstop, not an answer.
3. Shim scope. "Small and stateless" is the floor. A root listing from the Kubernetes API adds RBAC and watches.
4. Master as a single point of failure. Run a highly available master, or accept that its outage stops every read.
5. Reuse across campaigns needs a persistent tier. A memory tier cannot hold a corpus between campaigns months apart. Repeat-read savings need a disk tier sized to the corpus. Without one, Variant B's benefit is prefetch and node locality only.

Variant B also narrows egress. Campaign pods talk only to the bucket: images come through the local mount and models through the cache volume. The shim is the only component that talks to the IIIF origin.

Adoption spike

One to two days on a CPU-only cluster, before choosing Variant B:

If the metadata check fails, Variant B falls. Variant A needs no change to the wrapper's design, since the wrapper keeps its download stage.

Choosing

• Variant A is a small fraction of the operating surface and solves prefetch and origin shielding. If the evidence shows idle GPUs or origin load, start here.
• Variant B adds node-local reads, declarative warming, placement near the data, and a data layer other pipelines can share. It costs operating Fluid, Alluxio and the shim. Reach for it only if the data layer will be shared beyond this system, or if origin load demands persistent caching at corpus scale.