Gigapixel tile pyramid generation (DZI, IIIF, Google Maps, Zoomify)

Goal

Single top-to-bottom pass tile pyramid generation from zenpipe's streaming strip pipeline, with bounded RAM regardless of image height. Covers DZI (OpenSeadragon), IIIF Image API 3.0, Google Maps XYZ, and Zoomify tile layouts.

Target: process 100MP+ images with ~80-200 MB RAM for pyramid buffers; handle gigapixel (100K+ px wide) with <1 GB.

---

Core: TilePyramidSink

New Sink implementation. Receives full-width strips from upstream, manages pyramid levels, writes tiles.

Algorithm (mirrors libvips dzsave)

1. Per pyramid level: strip buffer holds width × (tile_height + 2×margin) pixels
2. When a level's strip buffer fills → write that level's tile row → 2×2 shrink into next level's buffer
3. Recursive cascade: one source strip can trigger tile writes at every pyramid level
4. Overlap carry-forward: margin pixels from bottom of strip copied to top of next strip position
5. All levels generated in a single top-to-bottom pass

Memory budget (RGBA8, 256px tiles, geometric sum ≈ 2× top level)

Image width	Pyramid buffers
10,000 px	~20 MB
40,000 px	~80 MB
100,000 px	~200 MB

2×2 box shrink

Trivial (a+b+c+d+2)>>2 over 2×2 pixel blocks. Not zenresize (which handles arbitrary ratios). Needs:

• Alpha-aware premultiply averaging
• Methods: mean (default), median, mode, nearest, max, min
• Odd dimension handling (replicate last row/column)
• SIMD via archmage (~100-200 lines)

---

Output layouts

Each layout is ~100-150 lines of path generation + metadata file.

DZI (Deep Zoom Image)

• Path: {name}_files/{level}/{col}_{row}.{ext}
• Descriptor: {name}.dzi (XML)
• Tile size: 254 default, overlap: 1
• Level 0 = 1×1 pixel, highest = full res

IIIF Image API 3.0 (Level 0)

• Path: {id}/{x},{y},{w},{h}/{tw},{th}/0/default.{ext}
• Descriptor: info.json
• Tile size: 512 default
• Region coordinates in full-resolution pixels, scaled by level
• Scale factors: powers of 2

Google Maps XYZ

• Path: {z}/{y}/{x}.{ext}
• Tile size: 256, no overlap
• Image padded to complete tiles (no partial/edge tiles)
• Blank tile skipping (configurable threshold)

Zoomify

• Path: TileGroup{n}/{level}-{col}-{row}.{ext}
• Descriptor: ImageProperties.xml
• Sequential tile numbering across all levels, grouped by 256

Output targets

• Filesystem (primary)
• Zip container (single file for S3/CDN deployment)
• Tiled TIFF / pyramidal TIFF (via image-tiff TiledImageEncoder, see feat/tiled-writing branch)
• PMTiles (future — Hilbert-curve tile IDs, directory compression, FNV-64a deduplication)

---

Parallel tile encoding

Biggest perf gap vs libvips. Each tile row may contain 100+ independent tiles.

• Per-tile JPEG/PNG/WebP encoding is embarrassingly parallel
• Scoped threads (rayon or manual) within TilePyramidSink::consume()
• No upstream pipeline changes needed — parallelism is purely inside the sink
• 40,000px wide / 256px tiles = 156 tiles per row → ~10× speedup with 16 threads

Optional double-buffered I/O: encode tile row N into buffer A, write A to disk while encoding row N+1 into buffer B.

---

Tile format encoding

Each tile encoded independently. Working format in the sink should be RGBA8 (not F32) — 4× less memory, tiles are always 8-bit output.

• JPEG (primary, via zenjpeg — "direct" encode from pixel region, no pipeline-per-tile overhead)
• PNG (for transparency/lossless)
• WebP (modern viewers)
• AVIF (future — encoding cost may be prohibitive per-tile)

Blank tile detection: hash tile against background color, skip if within threshold. Significant savings for images with uniform regions.

---

Gigapixel input: mmap + tiled TIFF

For gigapixel sources (COG, pyramidal TIFF), the input side needs tiled random access.

mmap as the universal input strategy

zencodec's existing Cow<'a, [u8]> API already supports mmap via Cow::Borrowed(&mmap[..]). No trait changes needed. On 64-bit (99% of deployments), address space is 256 TB+ — even 100 GB COGs map trivially.

zentiff tiled access (codec-specific, not zencodec trait changes)

// Accessed via DecodeJob::extensions() downcast
pub trait TiledContainerAccess {
    fn levels(&self) -> &[LevelInfo];
    fn raw_tile(&self, level: u32, col: u32, row: u32) -> Option<&[u8]>;
    fn tile_compression(&self, level: u32) -> Option<Compression>;
}

Enables:

• Raw JPEG tile passthrough (zero decode, zero encode — IIPImage-style serving)
• Pyramidal level selection (skip generating levels that already exist)
• Tile-row streaming into the pipeline (reassemble tile row → strip)

MmapTiledSource

New Source impl wrapping mmap'd tiled TIFF. Reads tiles left-to-right, top-to-bottom, reassembles into strips.

---

Analysis barriers without full materialization

For content-adaptive operations (auto-levels, CLAHE, saliency) that need two passes:

Strategy: always temp-file materialize

• Decode once, write pixels to temp file (tempfile + memmap2, or memfd_create on Linux)
• mmap the temp file back as a MaterializedSource
• OS page cache manages physical RAM — only hot pages resident
• Write cost: ~0.4s per GB to tmpfs, never the bottleneck
• Works for every codec regardless of decode cost
• No need for RewindableDecode trait or DecodeCost categorization

For cheap codecs (JPEG, uncompressed TIFF) with mmap'd input, constructing a fresh decoder from the same &[u8] is also viable — but temp-file is simpler and universally correct.

---

zencodec changes (minimal)

1. with_resolution_level_hint(level: u32) — new default-impl hint on DecodeJob. Codecs with pyramids (TIFF, JP2, JXL progressive) honor it; others ignore. Same pattern as with_crop_hint.
2. That's it for traits. Tiled access is codec-specific via extensions() downcast.

---

Cross-tile dependencies in column-parallel mode (future)

For extreme widths (>200K px), full-width strips become expensive. Column-parallel processing splits the pipeline into independent tile-width columns with overlap.

Halo computation

Each spatial operation declares its overlap (halo) requirement:

• Resize Lanczos3: ~3 px
• Gaussian blur σ: ~3σ px
• Clarity σ=20: ~60 px
• Per-pixel ops: 0 px

Halos compound through the pipeline. Graph compiler walks backward, accumulating and scaling through resize nodes.

Strategy per dependency type

Dependency	Solution
Spatial kernels (resize, blur, sharpen)	Halo overlap — redundant computation at column edges
Analysis (auto-levels, CLAHE)	Materialization barrier — full-width pass, then column-parallel
Compositing (overlay)	Clip to column bounds — each column handles its portion
2×2 pyramid shrink	Needs pixel pairs at column boundaries — merge before shrink

Column-parallel is a future optimization layered on top of the core tile sink. Ship full-width first.

---

image-tiff prerequisites

Prototype on feat/tiled-writing branch (611 lines, all tests pass):

• TiledImageEncoder — write NxN tiles with TileOffsets/TileByteCounts
• write_tile_precompressed() — raw byte passthrough (JPEG-in-TIFF)
• new_subfile_type() — mark overview IFDs (NewSubfileType tag)
• COG byte ordering (metadata-first IFD placement) — not yet implemented

---

Implementation order

1. TilePyramidSink + 2×2 shrink (~700 lines) — core algorithm
2. DZI + IIIF3 layout writers (~300 lines) — most common outputs
3. Parallel tile encoding (~300 lines) — biggest perf impact
4. Google Maps + Zoomify layouts (~200 lines) — complete the set
5. MmapTiledSource for zentiff (~300 lines) — gigapixel input
6. Temp-file MaterializedSource (~150 lines) — analysis barriers
7. Zip container output (~200 lines) — deployment convenience
8. Column-parallel execution (future) — extreme widths

Start by reading zenpipe's existing Sink and Source APIs, then inspect the image-tiff feat/tiled-writing branch and its tests. The first milestone is a bounded-memory TilePyramidSink with 2×2 shrinking and a tested output layout; the issue describes several later layouts, input sources, and performance features that would need separate scope.