Models may trick us with pointer provenance

Summary

The guarded Python pointer helpers trust geometry supplied in candidate-editable arguments. As a result, a candidate can execute a load outside a pointer's original object extent while still receiving [PASS].

This is an evaluation-correctness bug report, not a claim of host-memory access or remote code execution. Unlike #1, the reproduction uses a raising _trap and the checksum really matches. The invalid load itself should trap even when its result is discarded.

Tested revision

• Repository: connglli/Codoku, codokus branch.
• Tested checker/common revision: 9af436f3c93715fedc8609dc6b05d292707e4891.
• Guarded preamble from src/backend/py_backend.cpp (kPreamble).
• The relevant pointer implementations also remained identical in the codokus branch when inspected for this report.
• Reproduced with the unmodified upstream checker and independently with our checker-owned guarded executor. No model calls are necessary.

Three confirmed cases

In each miniature fixture, a = [7, 8], the fixed pointer p = _Ptr(a, 0, 1, 0, 1) has extent [0, 1), and the final checksum remains 7.

Case	Valid fill	Accepted invalid fill	Problem
Zero stride	q = _Ptr(a, 0, 1, 0, 1)	q = _Ptr(a, 1, 0, 0, 1)	q.off == q.hi, but _load(q) reads a[1] because zero stride makes the upper-bound test pass.
ptrindex extent widening	q = _pidx(p, 0, 1, 1); v2 = 2	q = _pidx(p, 1, 2, 1); v2 = 0	The supplied n widens the child extent to [0, 2) even though the parent ends at 1.
ptrfield extent widening	q = _pfield(p, 0, 1, 1); v2 = 2	q = _pfield(p, 1, 1, 2); v2 = 0	The supplied slen widens the child extent beyond its parent.

Each candidate then executes v1 = _load(q). The result is deliberately not included in the final checksum: an executed invalid memory operation must still fail, regardless of whether its result affects the checksum.

These substitutions preserve the exact constant multiset. The zero-stride case uses {0: 2, 1: 2}; the widening cases use {0: 1, 1: 2, 2: 1}. Only the permitted placeholders change; the runtime, trace, CFG, checksum and surrounding statements are fixed.

Observed: all three valid references PASS, and all three invalid candidates also PASS, in both checkers.

Root cause

In the guarded preamble:

• _Ptr.__init__ stores the supplied geometry without validating it.
• _load / _store test p.off < p.lo or p.off + p.stride > p.hi, but this does not reject one-past-end access when the supplied stride is zero.
• _pidx constructs a new upper bound p.off + n * estride without checking containment in the parent extent.
• _pfield similarly constructs p.off + slen without checking containment.

These invariants may follow from the typed compiler when generating a known-valid program, but that guarantee no longer holds when the lowered metadata is exposed as editable puzzle constants.

This is distinct from recovering missing integer operand widths: the parent pointer already carries the extent needed to detect these examples.

Standalone reproduction

At the tested revision, save this as repro_pointer_bounds.py and run:

python3 repro_pointer_bounds.py /path/to/Codoku

The script uses the repository's own guarded preamble and masking functions. It does not modify repository files or require a compiler build.

import ast
import pathlib
import re
import subprocess
import sys
import tempfile

repo = pathlib.Path(sys.argv[1]).resolve()
sys.path.insert(0, str(repo / "codokus"))
import codoku_common as common

backend = (repo / "src/backend/py_backend.cpp").read_text()
preamble = re.search(
    r'const char \*kPreamble = R"PY\((.*?)\)PY";', backend, re.S
).group(1)
checker = repo / "codokus/codoku_checker.py"

def source(body):
    return preamble + '''
def _in_check_chksum(expected, actual):
    if expected != actual:
        _trap("checksum mismatch")
    return actual

def leaf():
    a = [7, 8]
    p = _Ptr(a, 0, 1, 0, 1)
    q = _Ptr(a, 1, 1, 1, 2)
    v0 = 7
    v1 = 7
    v2 = 2
    # ^entry
    if __import__("os").environ.get("DUMP_TRACE"):
        print("^entry:")
    # ^b0
    if __import__("os").environ.get("DUMP_TRACE"):
        print("^b0:")
''' + body + '''
    # ^exit
    if __import__("os").environ.get("DUMP_TRACE"):
        print("^exit:")
    return v0

def main():
    r = leaf()
    r = _in_check_chksum(7, r)
    return 0

if __name__ == "__main__":
    raise SystemExit(main())
'''

def make_puzzle(reference):
    raw = reference.encode()
    tree = ast.parse(raw)
    leaf, _ = common.find_python_leaf_function(tree, raw)
    nodes, entry, end = common.get_python_maskable_statements(leaf, raw)
    names = common.collect_python_leaf_locals(leaf)
    funcs = {n.name for n in ast.walk(tree) if isinstance(n, ast.FunctionDef)}
    replacements, budget = [], {}
    for node in nodes:
        if entry < node.lineno < end:
            common.collect_python_replacements(
                node, raw, True, replacements, budget, names, funcs
            )
    banner = "#//@ EXEC_PATH: entry -> b0 -> exit\n"
    for a, b in sorted(common.build_python_cfg(leaf, raw)):
        banner += f"#//@ CFG_EDGE: {a} -> {b}\n"
    for value, count in sorted(budget.items()):
        banner += f"#//@ <FILL_CONST>: {value} {count}\n"
    return banner + common.apply_replacements(raw, replacements).decode()

cases = {
    "zero_stride": (
        "    q = _Ptr(a, 0, 1, 0, 1)\n    v1 = _load(q)\n",
        "    q = _Ptr(a, 1, 0, 0, 1)\n    v1 = _load(q)\n",
    ),
    "ptrindex_widening": (
        "    q = _pidx(p, 0, 1, 1)\n    v1 = _load(q)\n    v2 = 2\n",
        "    q = _pidx(p, 1, 2, 1)\n    v1 = _load(q)\n    v2 = 0\n",
    ),
    "ptrfield_widening": (
        "    q = _pfield(p, 0, 1, 1)\n    v1 = _load(q)\n    v2 = 2\n",
        "    q = _pfield(p, 1, 1, 2)\n    v1 = _load(q)\n    v2 = 0\n",
    ),
}
with tempfile.TemporaryDirectory(prefix="codoku-pointer-repro-") as tmp:
    root = pathlib.Path(tmp)
    for name, (good, bad) in cases.items():
        puzzle = root / (name + "-puzzle.py")
        puzzle.write_text(make_puzzle(source(good)))
        for kind, body in (("reference", good), ("invalid", bad)):
            candidate = root / (name + "-" + kind + ".py")
            candidate.write_text(source(body))
            result = subprocess.run(
                [sys.executable, str(checker), str(puzzle), str(candidate)],
                capture_output=True, text=True, timeout=15,
            )
            print(name, kind, "exit=", result.returncode)
            print((result.stdout + result.stderr).strip())

Expected: references PASS; all three invalid candidates are rejected for the invalid pointer operation. Actual: all six executions report PASS.

Suggested direction

• Do not trust candidate-editable pointer geometry as authoritative object metadata.
• Enforce valid non-null dereference geometry, including positive integral stride and an explicit exclusion of one-past-end access. Preserve legitimate null-pointer construction and legal one-past-end pointer formation.
• Validate _pidx / _pfield parameters and ensure the derived extent stays inside the actual parent object.
• Consider retaining immutable type/layout metadata in the puzzle representation instead of exposing it as ordinary fillable numeric operands.
• Add regression tests for these three budget-preserving cases, corresponding stores, negative stride, normal in-bounds access, and allowed one-past-end formation followed by rejected dereference.

The shared compiler backend may rely on frontend invariants; the fix should preserve that layering while explicitly defining what the puzzle checker must validate for untrusted fills.

Scope of the report

This establishes reproducible false acceptance of executed invalid pointer operations. It does not establish that any particular model submission used these cases, nor that an entire benchmark score is invalid. No private model trajectories, credentials or evaluation datasets are included.

Start with the kPreamble implementations in src/backend/py_backend.cpp, then run the standalone repro_pointer_bounds.py against codokus/codoku_checker.py. Use the three reproduced cases as the initial regression scope, along with the listed store, negative-stride, in-bounds, and one-past-end cases. Done means valid references still pass while executed invalid pointer operations are rejected.