PSNR ceiling at high bitrates caused by the non-orthogonal transform pair, with an encoder-only fix

## Summary

PSNR saturates at high bitrates: beyond a certain point, spending more bits does not raise PSNR, unlike other codecs. The cause is not the internal bit depth, the transform shifts, or the quantizer — it is the **non-orthogonality of the integer transform matrix pair**. An encoder-only change that is fully bitstream-compatible removes the ceiling; measurements below.

## Analysis

Ruled out by measurement (8x8 kernel harness, 20k random full-range blocks per case):

| hypothesis | measurement | verdict |
|---|---|---|
| coefficient grid (s16) too coarse | grid-only floor is ~83 dB | not the cause |
| forward-transform intermediate rounding | single-rounding forward changes nothing (0.4 dB) | not the cause |
| quantizer step / level clipping | at QP 0 max level 8953, zero clipped levels | not the cause |
| implementation bug | exact-arithmetic ITX reproduces the same floor | not the cause |

The floor comes from the matrix pair itself. For the 8x8 base matrix `T`, `T*T^T` has diagonal `32768, 32740, 33124, 32740, ...` and off-diagonal entries up to 50 — rows 2 and 6 (84, 35, ...) have a squared norm 1.1% larger than the others. The decoder applies `T^T` as the inverse, but `T^T != T^-1 * 32768`, so the reconstruction error is **proportional to the signal amplitude**: with full-range random blocks the round-trip is ~52 dB at any bit depth; at 1/8 amplitude it is ~69 dB; at 1/64 amplitude it is lossless.

This also explains why codecs sharing this matrix family do not show the ceiling in practice: they transform small prediction residuals, so the multiplicative error is proportionally small. APV has no pixel-domain prediction — full-amplitude samples enter the transform, exposing the floor in the working range.

## Encoder-only fix (bitstream compatible)

The forward transform is an encoder choice; only the inverse is normative. Replacing the forward matrix with the **least-squares pseudo-inverse of the normative inverse transform**, `G = (T*T^T)^-1 * T`, makes the encoder produce the coefficients that the fixed decoder reconstructs best. Scaled to s16 constants (2^19, i.e. 16x the base scale; entries are the base values x16 with small corrections, e.g. row 2: 1344 -> 1330) the matrix keeps its even/odd symmetry, so the same butterfly structure applies with a generic 4-multiply stage and +4 on both forward shifts.

Branch: https://github.com/AcademySoftwareFoundation/openapv/tree/improve_forward_transform

## Measurements

Kernel round-trip (transform + spec integer ITX): 52.0 -> 100.7 dB (10-bit), 52.0 -> 76.6 dB (12-bit, coefficient-grid bound).

End-to-end, pattern y4m 320x240 10-bit, medium preset, 3 frames:

| QP | size (base -> new) | PSNR (base -> new) |
|---|---|---|
| 0 | 88230 -> 88125 | 67.33 -> **87.46** |
| 2 | 83847 -> 83776 | 67.35 -> 83.96 |
| 6 | 76751 -> 76671 | 67.05 -> 78.15 |
| 12 | 66781 -> 66746 | 66.07 -> 71.87 |
| 18 | 58195 -> 58172 | 63.90 -> 66.30 |
| 24 | 50514 -> 50487 | 59.52 -> 60.16 |
| 30 | 43228 -> 43209 | 54.38 -> 54.52 |
| 40 | 32373 -> 32366 | 45.03 -> 45.07 |

The bitrate is unchanged (marginally smaller), the gain appears exactly where the ceiling used to bind, and mid/high QPs are untouched. 12-bit behaves the same (QP 0: 64.7 -> 79.7 dB, measured with the fix from #254 applied to the decoder). All ctest cases pass, including the conformance hash checks, since the decoder is not modified.

## Remaining work

- The branch forces the C forward transform; the AVX2/NEON `txb` kernels need the new constants (s16, madd-friendly) and the +4 shifts before this can be a PR
- BD-rate measurement on natural content to confirm the mid-QP neutrality
- The RDOQ distortion model keeps working unchanged since the coefficient scale is identical

Start with the improve_forward_transform branch and inspect the C forward transform alongside the AVX2/NEON txb kernels, which still need corresponding work. Run the existing ctest cases and compare the listed kernel and end-to-end PSNR measurements; completion also requires BD-rate results on natural content while preserving decoder conformance hashes.

Written by the indexing model from the issue text.