Auto-deskew: content-adaptive rotation angle detection

Goal

Implement content-adaptive auto-deskew so callers can straighten scanned documents, crooked photos, etc. without knowing the exact rotation angle in advance. The detected angle feeds into the existing rotate_angle() effect pipeline.

Current state

Infrastructure already in place:

• `DimensionEffect` trait in `zenlayout/src/dimension.rs` — `forward()` / `inverse()` return `Option<(w, h)>` specifically to support content-adaptive effects that don't know their output dimensions until pixels are analyzed.
• `RotateEffect` (same file) — accepts an angle and `RotateMode` (InscribedCrop / Expand / CropToOriginal).
• Non-cardinal rotation execution path (being wired up separately — see companion issue).

What's missing: the analyzer that determines the angle from pixels.

Design

API shape

Add a new `AutoDeskewEffect` implementing `DimensionEffect`:

```rust
pub struct AutoDeskewEffect {
pub mode: RotateMode,
pub max_angle: f32, // e.g., 10° — don't detect beyond this
pub method: AutoDeskewMethod,
}

pub enum AutoDeskewMethod {
/// Edge-density Hough transform — best for documents with clear horizontals
Hough { min_confidence: f32 },
/// Gradient moment — faster, works for photos with weak dominant lines
GradientMoment,
/// Horizontal projection variance — document text-line detection
ProjectionVariance,
}
```

`forward()` / `inverse()` return `None` — dimensions depend on detected angle.

Resolution phase

Add an `analyze()` method on `DimensionEffect` that takes pixel data:

```rust
trait DimensionEffect {
/// Content-adaptive effects override this; fixed effects use the default
/// that returns the static angle/parameters.
fn analyze(&self, _pixels: &[u8], _w: u32, _h: u32, _stride: usize)
-> Box {
self.clone_box()
}
}
```

The execute pipeline calls `analyze()` on the materialized frame before applying the effect. `AutoDeskewEffect::analyze()` returns a concrete `RotateEffect` with the detected angle.

Algorithm: Hough (recommended primary)

1. Convert to grayscale (Rec.709 luma).
2. Edge detect (Sobel).
3. Hough transform over the range [-max_angle, +max_angle] — parameterize lines by (ρ, θ).
4. Find dominant θ from the accumulator.
5. Return θ as the rotation angle to apply.

For efficiency on large images:

• Downsample to ~1000px on the long edge before analysis (angle is scale-invariant).
• Use gradient-magnitude-weighted voting instead of binary Canny.
• Restrict θ search to a narrow band around 0° (scan in 0.1° steps from -max to +max).

Algorithm: Gradient moment (alternative, faster)

Compute principal gradient direction via structure tensor:

```
J = Σ [Ix² IxIy]
[IxIy Iy²]
```

The eigenvector of J corresponding to the smaller eigenvalue points along the dominant edge direction. Angle = atan2(v.y, v.x). Cheap (O(N), one pass) but biased toward high-contrast photo content rather than text lines.

Algorithm: Projection variance (text documents)

For a text document, the optimal rotation maximizes the variance of horizontal line projections (each text row becomes a dense horizontal stripe, gaps become empty). Scan θ from -max to +max, compute variance of row-sum for each θ, pick argmax. More expensive than Hough for wide angle ranges but more robust for clean documents.

Implementation plan

Phase 1 — Gradient moment (small, fast)

• Add `AutoDeskewEffect` + `GradientMoment` method.
• Implement structure tensor on u8 luma.
• Wire `analyze()` through `execute_layout.rs`.
• Test on rotated document corpus.

Phase 2 — Hough

• Add Hough method with configurable angular resolution.
• SIMD-optimize the accumulator voting (per-angle rows).
• Compare accuracy vs gradient moment on the corpus.

Phase 3 — Projection variance

• Add for text-document special case.
• Auto-select method: if image is predominantly bi-level/low-entropy → projection; else Hough.

Test corpus

Needed before we start:

• Rotated scans (known GT angle): generate by applying known rotations to clean documents.
• Rotated photos: crooked horizons from a consumer photo corpus.
• Edge cases: uniform images (should return 0°), images with no dominant lines, extreme angles (±15° and beyond).

Acceptance criteria

• `AutoDeskewEffect` implementing `DimensionEffect` in zenlayout.
• Analyzer in zenpipe (or new `zendeskew` crate if heavy).
• RIAPI key `autodeskew=1` or similar.
• End-to-end test: decode → detect → rotate → encode produces a visibly straightened result within ±0.2° of ground truth on the test corpus.
• Mean detection time <50ms for a 4000×3000 image on the analysis downsample.

Related

• Companion issue: DimensionEffect execution (wire non-cardinal rotation through the graph).
• RIAPI gap: `autodeskew` is not currently in the legacy `Ir4Expand` or zen-native path — this would add a new key.

Start by reading zenlayout/src/dimension.rs and execute_layout.rs, along with the companion DimensionEffect execution issue. Begin with the Phase 1 GradientMoment plan and establish the rotated document and photo test corpus described here. Done means the analyzer is wired through decode-to-encode, exposes the RIAPI key, and meets the stated accuracy and timing criteria.