A helpful (???) guide for LLMs?

### Comment:

I've been working with LLMs on helping me with the hardest compilation issues at conda-forge.

This might be helpful for other users to include in the prompt.

```
# Field guide: debugging pytorch-cpu-feedstock (for future Claude/maintainer sessions)

Distilled from the v2.12.0 effort (PR #503). Read this before diving in — it will
save you hours. Companion principles came from the tensorflow-feedstock CLAUDE guides.

## 0. Orientation

- This feedstock is **conda-build** (classic `recipe/meta.yaml`), **not** rattler-build.
So the rattler-build `debug` workflow from the TF guide does **not** apply; the
conda-build analogue is `python build-locally.py --debug` / `conda debug`
(gated by `BUILD_WITH_CONDA_DEBUG=1` in `.scripts/build_steps.sh`).
- Outputs: `libtorch` (C++), `pytorch` (python), `pytorch-tests`, `pytorch-cpu`/`pytorch-gpu`.
`recipe/build.sh` is the unix build script for both `libtorch` and `pytorch` outputs and
branches on `$PKG_NAME`. `bld.bat` is Windows.
- A full build is the long pole (~1–2 h CPU, ~2 h CUDA on a laptop). **Build once, then
iterate on the failing phase** — do not re-run `build-locally.py` from scratch per change.

## 1. The fast local iteration loop (do this first)

The single biggest time-saver: a **persistent docker container** + **ccache** + a big disk.

```sh
# config = a filename (no .yaml) from .ci_support/, e.g.:
# linux_64_blas_implgenericc_stdlib_version2.17channel_targetsconda-forge_maincuda_compiler_versionNoneis_rcFalse (CPU)
# linux_64_blas_implgenericc_stdlib_version2.17channel_targetsconda-forge_maincuda_compiler_version12.9is_rcFalse (CUDA 12.9)
IMAGE=quay.io/condaforge/linux-anvil-x86_64:alma9 # from the config's docker_image
FS=$PWD # feedstock root

docker run -d --name pt -v "$FS":/home/conda/feedstock_root:rw,z,delegated \
-v "$FS/recipe":/home/conda/recipe_root:rw,z,delegated \
-v /BIG/build_artifacts:/home/conda/feedstock_root/build_artifacts:rw,z \ # keep off / (huge)
-e CONFIG=$config -e CONDA_FORGE_LOCAL_CCACHE=1 -e CCACHE_MAXSIZE=60G \
-e HOST_USER_ID=$(id -u) -e IS_PR_BUILD=True -e UPLOAD_PACKAGES=False \
-e CPU_COUNT=8 -e MAX_JOBS=8 "$IMAGE" sleep infinity
docker exec -u $(id -u) pt bash -lc 'micromamba install -p /opt/conda -y -c conda-forge ccache'

# Run the build (NB: must activate base — docker exec bypasses the image entrypoint, so
# the conda-forge-ci-setup helpers like setup_conda_rc are not on PATH otherwise):
docker exec -u $(id -u) -e flow_run_id= -e remote_url= -e sha= -e CI= pt bash -lc '
source /opt/conda/etc/profile.d/conda.sh; conda activate base
cd /home/conda/feedstock_root
bash .scripts/build_steps.sh > build_artifacts/build.log 2>&1'
```

Then iterate inside the **same** container without rebuilding from scratch.

**ccache hook (local only — NOT in the recipe):** to use ccache, add a small block to
`build.sh` (right after `CMAKE_BUILD_TYPE=Release`), gated on `CONDA_FORGE_LOCAL_CCACHE` so it
is inert in CI:
```sh
if [[ -n "${CONDA_FORGE_LOCAL_CCACHE:-}" ]]; then
CCACHE_BIN="$(command -v ccache || echo /opt/conda/bin/ccache)"
export CMAKE_C_COMPILER_LAUNCHER="$CCACHE_BIN" CMAKE_CXX_COMPILER_LAUNCHER="$CCACHE_BIN" CMAKE_CUDA_COMPILER_LAUNCHER="$CCACHE_BIN"
fi
```
Keep it in your working tree but **never** commit it (remove before `git add recipe/build.sh`,
re-add after). It is deliberately not committed.

**ccache gotchas:**
- The conda compiler activation overrides `CCACHE_DIR`; ccache ends up in the container's
`~/.cache/ccache`. Fine — it persists as long as the `sleep infinity` container lives.
- ccache hashes the **work dir path**, and conda-build re-extracts to a new timestamped
`build_artifacts/_/work` each run, so a *fresh* `conda-build` run mostly
**misses**. Set `ccache --set-config hash_dir=false` (and/or `base_dir`) to get cross-run
hits, or iterate the failing phase inside the existing work dir instead of re-running.

**Disk:** a CUDA build's `build_artifacts` is tens of GB; keep it on a big disk, not `/`.

**macOS / Windows have no docker.** `build-locally.py` runs `.scripts/run_osx_build.sh`
(native) on mac and `.scripts/run_win_build.bat` on Windows. On macOS, build once with
`python build-locally.py ` (it bootstraps a miniforge under `./miniforge3`),
then iterate in the produced `/conda-bld/_/work` dir. ccache works
natively (`brew install ccache` or the conda one) via the same `CMAKE_*_COMPILER_LAUNCHER`
hook. Windows uses `sccache` (already wired in `bld.bat`).

## 2. Diagnostic playbook (the methodology that worked)

- **Reproduce minimally before theorizing.** Build once, then run the single failing test in
isolation vs in the full suite. If it passes alone but fails in the suite → **test pollution**
(some earlier test leaks global state), not a real bug.
- **To find a polluting test:** append `pytest_runtest_setup`/`teardown` hooks to the test
dir's `conftest.py` (auto-loaded; `-p plugin` was unreliable here because pytorch's own
conftest interferes). Have the hook **write to a file** (pytest captures stdout) and probe
the suspected global state after each test; the first test that leaves it bad is the culprit.
- **For import/linking failures:** reproduce in the **driverless build container** (no `--gpus`)
— that mirrors CI. Use `readelf -d .so | grep NEEDED`, `nm -D --undefined-only`, and
`patchelf --print-needed` to see what's hard-linked vs undefined. The build log's
`Warning: Unused direct dependencies:` lines are gold.
- **Validate on the GPU too** before declaring a CUDA fix done: start a `--gpus all` container
(nvidia runtime / CDI) and run a real op (e.g. a `cuda` matmul). The 12.9 config builds
`compute_70+PTX` which JIT-runs on newer GPUs (e.g. sm_86), so GPU validation is possible
even on a non-listed card.
- **Always confirm via the canonical path** (`build_steps.sh` → conda-build's own test phase)
before pushing — manual patching of built libs proves the *idea*; the rebuild proves the
*recipe*.

## 3. Findings & fixes so far (v2.12.0 — pushed to hmaarrfk fork branch v2.12)

1. **CPU: `RuntimeError: element 0 of tensors does not require grad`** (789 failures).
*Not* an autograd bug. `test/test_custom_ops.py::test_incorrect_abstract_impl` enters
`_AutoDispatchBelowAutograd()` + `ExcludeDispatchKeyGuard` via raw `guard = ...; del guard`
in a `finally`; `opcheck()` is expected to raise, and the exception traceback pins the
`forward` frame so the C++ RAII guards aren't destroyed → the "exclude Autograd dispatch
key" thread-local leaks process-wide → every later op (even `nn.Linear`) loses its
`grad_fn`. **Fix:** convert the guards to `with` blocks (as a sibling test already does).
→ `recipe/patches/0019-...patch`, upstream **pytorch/pytorch#186550**. 789 → 0.

2. **Linux CUDA: `ImportError: libcuda.so.1: cannot open shared object file`.** ROOT CAUSE was
in **our own patch 0008**, which changed the correct upstream
`set(CUDA_NVRTC_LIB "${CUDA_nvrtc_LIBRARY}" CACHE FILEPATH "")` to
`get_target_property(CUDA_NVRTC_LIB CUDA::nvrtc INTERFACE_LINK_LIBRARIES)`. FindCUDAToolkit
defines `CUDA::nvrtc` with `DEPS cuda_driver`, so that property is `CUDA::cuda_driver`, **not**
libnvrtc. So `caffe2::nvrtc_runtime` linked the *driver* into `libtorch_cuda.so` instead of
libnvrtc, which caused ALL of: an **unused `libcuda.so.1` NEEDED** (driver is a soft,
system-provided dep, so `import torch` fails driverless — CI/CPU-only); **9 undefined
`nvrtc*` symbols** (cuDNN frontend); the **Windows `nvrtcCreateProgram` link failure**; and
`Failed to compute shorthash for libnvrtc.so`. **Fix (in patch 0008):** read
`IMPORTED_LOCATION` (the driver-free libnvrtc path) instead — one token, fixes all of the
above incl. Windows. Confirmed: `$PREFIX/lib/libnvrtc.so shorthash is …` (was "Failed to
compute"). Diagnosed via `readelf -d`/`nm -D --undefined-only` + the shorthash log line; an
earlier `patchelf` band-aid (commit 1dd2f72) was replaced by this source fix.

3. **macOS (`osx_arm64`, `osx_64`): RESOLVED by patch 0019 — same autograd leak as CPU.**
The pre-0019 osx_arm64 run failed with **372 failed, 9662 passed**, but *every one* of
those 372 was the same `_test_incorrect_abstract_impl` autograd-exclude-key leak from
finding 1: 334 `element 0 of tensors does not require grad and does not have a grad_fn`
plus the test_custom_ops leak-setup victims, spread across test_modules / test_nn /
test_linalg (`*_backward_*`) / test_custom_ops — **zero** genuine osx-specific failures
(no "Tensor-likes are not close" anywhere in the CI log). The `-march=core2` / MKL-discovery
worries from mgorny's old comments did **not** materialize for v2.12.0. With 0019, CI on
PR #509 is green for **all three** osx jobs (osx_arm64, osx_64 generic, osx_64 mkl).
*Local-build gotcha:* a native build on a recent Mac (macOS 26 / newer Apple Silicon)
shows ~4 extra `test_non_contiguous_tensors_nn_Transformer*_mps_float16`
"Tensor-likes are not close" failures. These are **MPS float16 precision deltas specific to
newer local hardware** — they do **not** reproduce on conda-forge's current CI runners
(CI passes without them). They are nonetheless added to the meta.yaml skip list as
**tactical skips** (grouped with the existing `*_mps_float16` precision skips) for
robustness across macOS/hardware versions. Caveat: this is deliberate over-skip territory
(cf. skip-hygiene PR #353) — if CI hardware/tolerances change, revisit. Local osx test runs
≠ CI; trust CI for osx pass/fail.

4. **Windows CUDA: `unresolved external symbol nvrtcCreateProgram`.** **Same root cause as
2(B)** — `torch_cuda` references nvrtc directly but nvrtc isn't linked. Not yet fixed;
the Linux `patchelf` fix doesn't translate. Fix in `bld.bat` (link nvrtc.lib) or, better,
the underlying CMake (`cmake/public/cuda.cmake`: `get_target_property(CUDA_NVRTC_LIB
CUDA::nvrtc INTERFACE_LINK_LIBRARIES)` yields empty/wrong here, so `caffe2::nvrtc_runtime`
links nothing). A CMake fix would resolve **both** Windows and Linux 2(B) and be upstreamable.

## 4. Principles (carry these forward)

- **`libcuda.so.1` / `libnvidia-ml.so.1` are the DRIVER = soft, system-provided.** Never
hard-link them and never symlink their stubs into `$PREFIX/lib`. A CUDA build must
`import` with no driver present. (From the TF guide: *"never put CUDA libs in LDFLAGS or
symlink their stubs into $PREFIX/lib."*)
- **When a failure is a genuine pytorch bug, fix it as an upstreamable git patch** (proper
message + `Signed-off-by`), drop it in `recipe/patches/00NN-...`, reference it in
`meta.yaml` with a `# backport ` comment. When it's a feedstock concern (e.g. the
`--as-needed` strip), fix it in `build.sh`/`bld.bat`.
- **Copy from sibling feedstocks, don't invent** (jaxlib, tensorflow, the cuda migrators).
- **Test pollution is common in pytorch's suite** — the recipe already skips
`test_base_does_not_require_grad_mode_*` etc. for the same reason. Prefer fixing the leak
upstream over adding skips.

## 5. Open items / next steps

- **macOS** (`osx_64`, `osx_arm64`) — **DONE.** All three osx jobs are green on CI (PR #509)
with patch 0019 alone; the 372 osx_arm64 failures were 100% the autograd leak, no genuine
osx-specific bug (see finding 3). Nothing left to do here unless CI regresses.
- **Windows** nvrtc (issue 4): implement the nvrtc link fix (see §3.4).
- **aarch64**: the Linux `patchelf` CUDA fix is `# [linux]` so it covers aarch64 too, but it's
unbuilt/untested locally and may carry extra aarch64-specific failures (x86 cpuid on aarch64).
- **Getting fixes onto the PR:** PR #503's head is `mgorny:v2.12`; fixes pushed to
`hmaarrfk:v2.12` are not on the PR until merged onto mgorny's branch.
- **`[TESTING]` commits** on the branch limit to py3.14 and a single SM — revert before final.

```