Add image comparison tests (proposal)

@a-hurst @mraspaud I had Claude put together a plan for adding image comparison based tests together. The full specification it created is below. What do you think about this? I had told it I was worried about having a lot of images sitting in the repository and having to change them over time. I also told it I was worried that issues sometimes aren't noticed unless large images are rendered. It suggested/recommended that using hashes as a quick check is a good test, but that having the references images would be good for tolerance checks and creating diff images. I wasn't completely sold on that idea until it pointed out that the test images even if most are 512x512 wouldn't be too large and it could store diff images as artifacts in the failing github CI making them easy to download and review. What do you guys think?

---

Image-based regression testing

Status: ready to implement. Nothing here has been built yet.
Verified against: a325dc1. Every file:line citation below was checked at that commit.

Why

The test suite has 38 tests and no whole-image coverage of any kind. It asserts isolated pixels
(aggdraw/tests/test_pen.py) or aggregate ink counts (aggdraw/tests/test_path.py), and
several tests call the API without asserting anything at all — test_draw.py::test_graphics
draws sixteen shapes and checks nothing, test_draw.py::test_transform calls settransform
five ways and checks nothing, test_symbol.py::test_symbol constructs eight symbols and checks
nothing. Whole Draw methods are untested: arc, chord, pieslice, rounded_rectangle,
setantialias, frombytes, text, textsize, along with every surface mode except "RGB".

The concrete consequence is recorded in specs/SPEC-04-agg2-audit.md, which is blocked on this
spec. An audit of the vendored agg2/ sources cannot start because there is no way to
demonstrate that a change under agg2/ left rendering untouched. SPEC-04 names the absence of
this harness as "still the real blocker".

The request that prompted this spec was for an Acid2-style approach — a small number of dense
images exercising most of the library — under two constraints: the repository should not
accumulate a large, churning pile of binaries, and the result must be reliable across
platforms, CPUs, and build environments.

What was decided

Question	Decision
Canvas size	512² default in mode L; RGB only where color is under test; one 1024² stress scene
What is stored	hashes.json as the primary check, plus one reference PNG per scene as the tolerance fallback and human-readable diff
Cross-platform strictness	Tolerance-passing mismatches warn everywhere; one CI job sets AGGDRAW_STRICT_IMAGES=1 and fails on them
Text	Included, using a font subset to printable ASCII (19 KB, not the full 740 KB face)
New dependencies	None. numpy, Pillow and pytest are already in the tests extra

Evidence

Everything in this section was measured against the build at a325dc1 unless marked otherwise.
Section "Reproducing the measurements" at the end gives the scripts. Claims derived from
reading the source, rather than from measurement, are marked (inferred) — the
implementation session must not treat those as settled.

Rendering is deterministic enough to hash

AGG's scanline rasterizer is genuinely integer fixed point. agg2/include/agg_rasterizer_scanline_aa.h:47-56
defines poly_base_shift = 8 and poly_coord(double c) { return int(c * poly_base_size); },
i.e. 24.8 fixed point on a 1/256 subpixel grid, and agg2/src/agg_rasterizer_scanline_aa.cpp
contains zero occurrences of float or double — cell accumulation, coverage, and sorting
are entirely integer. Coverage runs through a precomputed 256-entry integer gamma LUT, and
aggdraw only ever installs gamma_linear() (identity) or gamma_threshold(0.5)
(aggdraw/_aggdraw.cxx:388-394), so no pow is in the drawing path.

Shape coordinates are narrowed to float32 at the API boundary — GETFLOAT at
aggdraw/_aggdraw.cxx:744-747, and every shape entry point parses with "(ffff)ff". Path is
the exception and keeps double. That narrowing is IEEE-exact and therefore contributes no
cross-platform divergence.

Supporting precedent: test_symbol.py::test_graphics2 asserts an exact
np.asarray(image).sum() == 50800 and has survived every CI platform since 2017 (commit
1832993), through the AGG 2.4 port. It draws a straight line, so it evidences only the
integer-only tier — it says nothing about arcs.

Canvas size does not threaten stability, and buys sensitivity

The same picture was rendered at several sizes while perturbing every coordinate by a
relative amount. Relative is the right model: floating-point error scales with coordinate
magnitude, while AGG's subpixel grid stays fixed at 1/256, so a larger canvas is intrinsically
more exposed. Cell values are the number of pixels that changed.

canvas	rel 1e-12	1e-10	1e-9	1e-8	1e-7	1e-6
512²	0	0	0	1	55	283
1024²	0	0	0	5	883	1816
2048²	0	0	0	18	2071	5325
4096²	0	0	2	47	5261	19400

Realistic platform divergence — one ULP of a double in sin/cos, or an FMA contraction — is
around 1e-16. Even at 4096² nothing moves until 1e-9, leaving roughly seven orders of
magnitude of headroom. Canvas size is close to free on the stability axis.

Read the other way, the same table is the argument for going bigger: at a fixed 1e-7
perturbation a 512² scene shows 55 changed pixels and a 4096² scene shows 5261. Antialiased
edge pixels — the only pixels that can register a subtle change — scale linearly: 2,643 at
256², 5,290 at 512², 10,618 at 1024².

Why large images reproduce bugs that small ones do not

This is mechanical, not folklore. AGG allocates rasterizer cells in blocks of 4096
(agg2/include/agg_rasterizer_scanline_aa.h:84-88: cell_block_shift = 12,
cell_block_pool = 256, cell_block_limit = 1024). aggdraw/_aggdraw.cxx:436 and :447 call
rasterizer.reset() once per fill and once per stroke, so the cell count is per shape, not
per scene. A single shape must exceed ~4096 cells before allocate_block() is ever executed;
a stroked ellipse reaches that around a 1024px canvas. At 256² no test in the suite would ever
run that code path.

Both historical bug regressions in this repository used large canvases for exactly this reason:
issue #14 at 800×600 (test_symbol.py::test_graphics2), issue #22 at 480×1024
(test_pen.py::test_graphics3).

Tolerance calibration

Measured on a 256² curve-heavy scene. RMS is over all channels; max Δ is the largest
single-channel absolute difference.

change	RMS	max Δ	px differing
coordinate shift 1e-6 (far above any FP divergence)	0.000	0	0
coordinate shift 1e-4 (noise floor)	0.019	1	22
1/256 px (one subpixel unit)	0.176	3	1597
ink color (0,0,0) → (1,1,1)	0.208	1	2859
pen width 2.5 → 2.6 (smallest realistic regression)	2.213	22	2884
pen width 2.5 → 3.0	11.119	102	3589
0.25 px shift	9.581	112	3362
1 px shift	32.530	255	4696

Defaults of rms_tol = 0.5 and max_tol = 4 sit between the noise floor and the smallest
realistic regression, with roughly 4× margin on RMS and 5× on max Δ.

The (0,0,0) → (1,1,1) row is the important one: a genuine regression that lands below any
workable tolerance, indistinguishable from noise by RMS or max Δ. That is why the exact hash
must be the primary check and tolerance only ever a fallback. A tolerance-only design would
silently pass it.

Storage cost

Realistic line art, PNG with optimize=True:

canvas	mode L	mode RGB
256²	4.8 KB	11.0 KB
512²	9.7 KB	21.9 KB
1024²	19.5 KB	44.4 KB
2048²	40.5 KB	96.5 KB

The scene catalogue below lands near 150 KB total. Using mode L wherever colour is not the
subject under test is what keeps that number down — it roughly halves the per-scene cost.

Where floating point enters, and what that means per scene

All of the following is (inferred) from reading the sources; none of it has been observed
to actually diverge.

• cos/sin per vertex in agg2/src/agg_arc.cpp:70-77, reached by arc, chord,
  pieslice, ellipse, and rounded_rectangle. These are not correctly rounded and differ
  between glibc, musl, Apple libm, and the MSVC UCRT — and between glibc versions.
• FMA contraction. setup.py passes no extra_compile_args (the Extension( call at
  setup.py:156 sets only define_macros, include_dirs, library_dirs, libraries), so
  GCC and Clang default to -ffp-contract=fast. On the aarch64, macos-arm64 and win-arm64
  wheels FMA is baseline and contraction will happen, in agg_trans_affine, in the miter
  calc_intersection, and in the Liang–Barsky clipper.
• The clipper is always live. aggdraw/_aggdraw.cxx:385 unconditionally calls
  rasterizer.clip_box(0, 0, xsize, ysize), so any geometry crossing a canvas edge runs
  clip_liang_barsky, which computes in double and truncates to int. Geometry entirely
  inside the canvas never touches it.
• Curve flattening has a truncation cliff. agg2/src/agg_curves.cpp:43 (and :151 for
  cubics) computes m_num_steps = int(len * 0.25 * m_scale) from a sum of sqrts. sqrt is
  correctly rounded, but the truncation to int means a len landing within an ULP of a
  multiple of 4 could yield a different step count and a visibly different polyline.
• Every fill runs through conv_contour (aggdraw/_aggdraw.cxx:430-437) with
  auto_detect_orientation(true), which decides orientation from calc_polygon_area(...) > 0.0.
  A near-zero-area polygon can flip orientation and invert the ±0.5px contour offset.
• Strokes avoid trigonometry. conv_stroke is left at its butt_cap + miter_join
  defaults and aggdraw never calls line_join/line_cap, so round joins and round caps —
  the only stroke paths that use calc_arc — are unreachable.

Two direct consequences for scene authoring:

1. Every scene marked strict must stay fully inside the canvas and must avoid zero-area,
   self-intersecting, and duplicate-consecutive-point polygons.
2. Symbol parsing uses strtod throughout (aggdraw/_aggdraw.cxx:2056 onward), which is
   LC_NUMERIC-dependent. Scenes assume the C locale; note this in the module docstring.

Text is a different problem

agg2/font_freetype/agg_font_freetype.cpp:441 defaults m_hinting(true) and :822 passes
FT_LOAD_DEFAULT on that basis. Glyph rasterization is done by FreeType itself, so output
depends on the TrueType bytecode interpreter version, the autohinter, and the FreeType
rasterizer — all of which change across FreeType releases. aggdraw exposes no way to disable
hinting.

Separately, Windows wheels are built with AGGDRAW_FREETYPE_ROOT='' (pyproject.toml), so
Draw.text, Draw.textsize and Font are compiled out entirely — they must be skipped,
not tolerance-compared.

For calibration, pycoast (the closest sibling project, also PyTroll, also an aggdraw consumer)
carries 5.1 MB of reference PNGs, and its baseline churn is almost entirely font-driven:
"Fix ref images for pillow 9.4", "Regenerate two_shapefiles_pil image with pillow 9.2.0",
"Update test images after vertical text position fix".

Design

Layout

aggdraw/tests/
  _scenes.py                  NEW   scene registry — the single source of truth
  test_scenes.py              NEW   parametrized over the registry
  test_text.py                NEW   FreeType-guarded property assertions
  regen_references.py         NEW   python -m aggdraw.tests.regen_references
  _helpers.py                 EDIT  comparison helpers; fix to_image for BGRA
  reference/
    hashes.json               NEW   {scene_name: sha256}
    <scene>.png               NEW   one per scene
  data/
    DejaVuSans-subset.ttf     NEW   ~19 KB
    DejaVu-LICENSE.txt        NEW

Scene registry (_scenes.py)

A frozen Scene dataclass and a decorator that registers into a module-level SCENES dict:

@dataclass(frozen=True)
class Scene:
    name: str
    draw_fn: Callable          # takes a Draw, draws on it, returns nothing
    mode: str = "L"
    size: tuple = (512, 512)
    background: object = 255
    strict: bool = True        # participates in exact-hash checking
    needs_freetype: bool = False
    rms_tol: float = 0.5
    max_tol: int | None = 4    # None means "not checked"
    note: str = ""             # why a scene is non-strict, or has a wide tolerance

plus render(scene), which builds the Draw, calls draw_fn(d), calls flush(), and returns
the Draw. Scene bodies only draw; the harness owns surface construction, so mode and size
stay declarative and uniform, and a scene can be re-rendered at a different size without
touching its body.

The registry is the single source of truth: test_scenes.py and regen_references.py both
iterate it, so a new scene cannot be added to one and forgotten in the other.

Comparison protocol (_helpers.py)

assert_scene_matches(scene, output_dir):

1. Render. raw = draw.tobytes(); h = sha256(raw).hexdigest().
2. If h == hashes[scene.name] → pass. No PNG is decoded — this is the fast path and the
   common case.
3. Otherwise load reference/<name>.png and compute RMS and max absolute difference with
   numpy.
4. If within rms_tol and max_tol:
   • AGGDRAW_STRICT_IMAGES is set and scene.strict → fail, reported as drift;
   • otherwise warnings.warn(...) naming the scene and both metrics → pass.
5. If outside tolerance → fail, after writing <name>.actual.png, <name>.expected.png
   and <name>.diff.png into output_dir and printing the paths in the assertion message.
6. If the hash entry or the reference PNG is missing → fail with a message pointing at
   python -m aggdraw.tests.regen_references.

output_dir is AGGDRAW_TEST_OUTPUT_DIR when set, otherwise pytest's builtin tmp_path
fixture — which needs no conftest.py, so the repository's current "no conftest" state is
preserved.

Everything uses numpy, which _helpers.py already imports at module level and which is already
in the tests extra. This design adds no new dependency.

_helpers.to_image needs an explicit-mode override before the modes_bgra scene can work: a
BGRA surface reports .mode == "RGBA", which the helper's own docstring already warns about.
Hashing operates on draw.tobytes() directly and so is unaffected; only the PNG side needs it.

Scene catalogue

phase	scene	size	mode	strict	covers
1	acid	768²	RGB	yes	Acid2-style: every shape method, several pens and brushes, opacity, a Path using all segment types, a Symbol
1	pen_widths	512²	L	yes	widths 0.5–12 on half-pixel coordinates, opacity ramp
1	paths	512²	L	yes	all eight Path methods, open vs closed, the degenerate-close case
1	arcs	512²	L	yes*	arc, chord, pieslice, ellipse; wrap-around and negative angles
1	cell_stress	1024²	L	yes	one path exceeding 4096 rasterizer cells, to reach allocate_block()
2	brush_fills	512²	RGB	yes	solid, opacity, 4-tuple-overrides-opacity, the colour-format matrix
2	rects	512²	L	yes	rectangle, rounded_rectangle across radii, polygon
2	symbols	512²	L	yes	Symbol operators M/L/C/S/H/V/Q/T/Z and the scale argument
2	transforms	512²	L	yes*	settransform: offset, offset+rotation, full 6-tuple
2	antialias_off	512²	L	yes	setantialias(False) over curves
2	clipped	512²	L	no	geometry crossing the canvas edge — Liang–Barsky runs in double
2	modes_{l,rgb,rgba,bgr,bgra}	256²	varies	yes	the same drawing per surface mode; BGRA bypasses to_image
3	text_basic	512²	L	no	Font + text at several sizes

* arcs and transforms start strict=True deliberately, despite the trigonometry and
FMA analysis above. Whether exact hashing survives for curve geometry is genuinely unknown; the
warn-everywhere design makes it safe to find out empirically. If the strict CI job reports
drift, demote the scene and record the observed platform and metrics in its note field. This
converts an unresolved question into data instead of a guess baked in up front.

acid is the Acid2 analogue and the canary — one hash covering most of the library. The
focused scenes exist because a failing acid hash localises nothing on its own.

Text tier

• Vendor a fontTools subset of DejaVuSans over printable ASCII with hinting preserved
  (aggdraw always renders hinted, so a de-hinted subset would not represent real output).
  Measured at 19 KB versus 740 KB for the full face. fontTools is a one-time authoring tool and
  must not become a test dependency; the subset font is committed. Ship DejaVu's license
  alongside it. Record the exact subsetting invocation in a comment in regen_references.py
  so the artifact is reproducible.
• text_basic is strict=False with a generous rms_tol (start at 6.0) and max_tol=None,
  because hinting can shift a stem by a whole pixel between FreeType versions.
• Pair it with property assertions in test_text.py that do not churn: ink lands inside the
  expected bounding box, the empty string draws nothing, textsize grows with font size, and
  textsize grows with string length. These carry the real regression-detection weight; the
  golden is a coarse backstop.
• Guard on hasattr(Draw, "text"), not try/except — on a FreeType-less build the methods do
  not exist at all.

Packaging

This is easy to miss and will silently break the wheel tests. pyproject.toml has no
package-data, and MANIFEST.in covers only LICENSE.txt and agg2/, so non-.py files
under aggdraw/tests/ are not packaged today. cibuildwheel runs
pytest --pyargs aggdraw.tests against the installed wheel, so the references must ship.

[tool.setuptools.package-data]
"aggdraw.tests" = ["reference/*.png", "reference/*.json", "data/*"]

plus matching recursive-include lines in MANIFEST.in for the sdist. Resolve both
directories through importlib.resources.files("aggdraw.tests") rather than __file__
arithmetic, so they resolve correctly from site-packages.

CI

In .github/workflows/ci.yml:

• Set AGGDRAW_STRICT_IMAGES: 1 on exactly one matrix entry — ubuntu-latest with one Python
  version. That job is the reference platform.
• Set AGGDRAW_TEST_OUTPUT_DIR and add an actions/upload-artifact step with if: failure(),
  so diff PNGs from a failed run are retrievable rather than lost with the runner.
• cibuildwheel jobs stay in warn mode. Per-job environment is awkward there, and the conda
  matrix already spans ubuntu/macos/windows.

Regeneration workflow

python -m aggdraw.tests.regen_references [--scene NAME] rewrites hashes.json and the
reference PNGs from the current build, and prints a reminder to look at the images before
committing. Document that regenerating is a deliberate act belonging in its own commit with an
explanation — never bundled into an unrelated change. Because hashes.json is text, the PR
diff shows one line per changed scene, which is the reviewable signal that a binary-only design
would not give.

Implementation phases

Phase 1 — harness and proof. _scenes.py, _helpers.py additions, test_scenes.py,
regen_references.py, packaging, CI wiring, and the five phase-1 scenes. This is the point at
which the design is validated and CI starts producing cross-platform drift data.

Phase 2 — coverage. The remaining non-text scenes, closing the gaps listed in "Why" above.

Phase 3 — text. Subset font, text_basic, test_text.py.

Optional items — each needs its own decision

These are adjacent, not part of the harness. Do not fold them in silently.

• -ffp-contract=off in setup.py's extra_compile_args. This removes the only concrete
  x86-versus-ARM64 divergence path — FMA fusion in agg_trans_affine, the miter intersection,
  and the clipper — at negligible cost. It is a build change, not a test change, and has
  not been decided. It would make strict hashing meaningfully more likely to hold on the ARM
  wheels.
• Expose the FreeType version from _aggdraw.cxx (the FREETYPE_MAJOR/MINOR/PATCH
  macros, or FT_Library_Version). That would let text goldens be keyed per FreeType version
  and strict-checked, instead of being permanently tolerance-only. Useful independently — there
  is currently no way to ask which FreeType a wheel was built against.
• test_pen.py::test_graphics3 is a truncated dead test. Its body ends at
  p = Pen((90,) * 3, 0.5): it creates an image and a pen, never draws, and asserts nothing.
  It is the issue #22 regression. Fold it into a real scene rather than leaving it.

Documentation to update

• AGENTS.md, "Testing conventions", currently states "No golden or reference images
  anywhere" and "No fixtures and no conftest.py". The first becomes false. Rewrite the
  section to describe the scene registry, the hash-plus-fallback protocol, and the regeneration
  workflow, and to say that new drawing features are expected to land with a scene. The
  conftest claim stays true — tmp_path is a builtin fixture.
• specs/SPEC-04-agg2-audit.md's entry criteria name this harness as the remaining blocker.
  Note there that SPEC-06 satisfies it.

Reproducing the measurements

Each table above comes from a short script over aggdraw + Pillow. The sensitivity table
renders one picture at several sizes with all coordinates scaled by (1 + rel) and counts
differing pixels:

def render(S, rel=0.0):
    def c(v):                      # v in 0..1 -> pixel coord, perturbed relatively
        return v * S * (1.0 + rel)
    d = aggdraw.Draw('L', (S, S), 255)
    p = aggdraw.Pen(0, 0.01 * S)
    d.ellipse((c(.08), c(.08), c(.70), c(.47)), p, aggdraw.Brush(0))
    d.arc((c(.12), c(.55), c(.95), c(.95)), 15, 300, p)
    d.line((c(.05), c(.95), c(.95), c(.05)), p)
    pth = aggdraw.Path()
    pth.moveto(c(.08), c(.70))
    pth.curveto(c(.25), c(.35), c(.55), c(.95), c(.70), c(.70))
    d.path(pth, p)
    d.flush()
    return Image.frombytes('L', d.size, d.tobytes())

Differences are counted with ImageChops.difference(a, b); RMS and max Δ come from
ImageStat.Stat(diff).rms and diff.getextrema(). The tolerance table applies the same
approach with the perturbation replaced by a pen width, colour, or offset change. The storage
table saves each render with Image.save(buf, 'PNG', optimize=True).

The implementation session should re-run these on its own build before trusting the specific
numbers, since they were taken on one Linux x86-64 build with Pillow 12.3.0.

Open questions

1. Does exact hashing actually hold for curves across platforms? The analysis says the
   margin is ~7 orders of magnitude; it has not been observed. The strict=True-by-default
   choice for arcs and transforms is designed to answer this from CI rather than assume it.
2. How often will text goldens need re-recording? pycoast suggests "every FreeType or
   Pillow bump". If phase 3 proves noisy, the fallback is to drop the text golden and keep only
   the property assertions in test_text.py.
3. Is 4096 cells the right target for cell_stress? The block size is known
   (cell_block_shift = 12), but the mapping from a drawn shape to a cell count is not exact.
   The implementation should verify the scene actually crosses the boundary — the cheapest
   check is to scale the shape up until rendering time or memory shows the discontinuity.