EnzymeAD / EnzymeAD/Enzyme-JAX
Building MFEM through the raising path: tracking the remaining blockers
- Dominant language
- MLIR
- Stars
- 131
- Forks
- 53
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 193
Description
Tracking issue for getting MFEM (`dfem-dev`) to build through the Reactant/EnzymeMLIR raising path, so it can be compared against the ClangEnzyme path.
Setup: MFEM wires AD in only through `find_package(Enzyme HINTS ${ENZYME_DIR})` + `target_link_libraries(mfem PUBLIC ClangEnzymeFlags)`, so a shim `EnzymeConfig.cmake` defining `ClangEnzymeFlags` swaps the whole toolchain with no MFEM edits. Each `-mllvm ` pair must be its own `SHELL:` group or CMake de-duplicates the repeated `-mllvm`.
Baseline for comparison: ClangEnzyme builds MFEM and passes `punit_tests "[dFEM]"` with 123 assertions in 23 cases.
## Fixed
| | |
|---|---|
| #2774 | `llvm.invoke` reached `CFGToSCF.cpp`, which rejects terminators with side effects. Blocked 14 files, none of which mention enzyme — the plugin raises **every** TU it is attached to, not only ones calling `__enzyme_autodiff`. Fixed with `LowerInvokePass` + `SimplifyCFGPass` before raising. Same PR guards `arith.bitcast` against changing shape. |
| #2775 | `isCallNonCapturing` resolved callees with an uncached `SymbolTable::lookupSymbolIn`, a linear scan of every module symbol, once per call per allocation. ~50% of `polygeist-mem2reg` runtime on a module with 1635 symbols / 1624 allocations / 11045 calls. Compile-time only. |
| #2776 | `polygeist-mem2reg` **did not terminate** on `mesh.cpp`. The transfer branch materializes a read of a memcpy's source; several early exits leave without anything asking for it. The orphaned read is a load of the other side of the transfer, so promoting that side sweeps it and counts the sweep as progress, and promoting this side writes it again next round. IR was byte-identical between rounds. 900s+ → **1.57s**. |
| #2777 | The index-cast rewrites in `CanonicalizeLoops.cpp` asked `index` for a width via `getIntOrFloatBitWidth`; unchecked in release, it reads index as a float and segfaults in `getFloatSemantics`. Six ops affected: `addi`, `subi`, `muli`, `shli`, `shrui`, `divui`. |
Each of these was masking the next — the hang had to be fixed before #2777 was reachable, and #2777 before the current failures were.
## Where the build stands
With all four on `main` (`23396e930`): **255 objects build, 16 fail**, so `libmfem` does not link and `punit_tests` cannot run yet.
**14 segfault**, all with one signature:
```
mlir::Type::getContext() <-- SIGSEGV
AffineApplyNormalizer::renumberOneDim(mlir::Value)
AffineApplyNormalizer::AffineApplyNormalizer(...)
composeAffineMapAndOperands(...)
fully2ComposeAffineMapAndOperands(...)
handle(PatternRewriter&, arith::CmpIOp, ...)
MoveIfToAffine::matchAndRewrite(scf::IfOp, ...)
AffineCFGPass::runOnOperation()
```
`fem/dgmassinv.cpp`, `fem/hybridization.cpp`, `fem/integ/bilininteg_diffusion_pa.cpp`, `fem/integ/bilininteg_hdiv_kernels.cpp`, `fem/integ/bilininteg_mass_pa.cpp`, `fem/qinterp/det.cpp`, `fem/quadinterpolator.cpp`, `fem/quadinterpolator_face.cpp`, `fem/tmop/assemble/{diag2,grad2_limit,grad3_limit}.cpp`, `fem/tmop/mult/{grad2_limit,mult2_limit}.cpp`, `fem/tmop/tools/energy2_limit.cpp`
One further file crashes in `MoveWhileToFor::matchAndRewrite` instead.
**2 fail with pass errors** rather than crashing — `fem/fe/fe_pos.cpp` and `fem/lor/lor_batched.cpp`:
```
error: division by non-positive value is not supported
error: 'affine.store' op operand cannot be used as a dimension id
note: see current operation: "affine.store"(%531, %111, %arg40, %104, %51, %41, %41)
<{map = affine_map<(d0)[s0, s1, s2, s3] -> (((-d0 + s2 + 1) * (-d0 + s3)) ...
```
## The `renumberOneDim` crash
The crash is on the last line of
```cpp
AffineDimExpr AffineApplyNormalizer::renumberOneDim(Value v) {
...
return cast(getAffineDimExpr(iterPos->second, v.getContext()));
}
```
`v.getContext()` is `v.getType().getContext()`, and it is `Type::getContext()` that faults.
**The Value is not null.** An instrumented null check on `v` never fired across the whole failing compile, so this is a **dangling** Value — a freed operation's result reaching the affine operand list — not a missing null check. `handle()` itself contains no `eraseOp`/`replaceOp`, so the stale operand most likely predates the call: either the greedy driver folded an op the condition still refers to, or `AffineApplyNormalizer` is retaining values across rewrites.
### Reproducing
The imported-module capture (`DEBUG_REACTANT_IMPORTED_MLIR_MOD_PATH`) is **not** sufficient here — replaying it stops earlier on the pass errors above. What works is dumping the enclosing `ModuleOp` from inside `MoveIfToAffine::matchAndRewrite` on every attempt; the last file written before the crash reproduces standalone:
```console
$ enzymexlamlir-opt --affine-cfg crash_min.mlir -o /dev/null
Segmentation fault
```
on a clean binary, in 1.17s. Function-level delta debugging takes it from 16 functions to **1** (1160 lines, `quadrature_interpolator::Det3D<0,0,false>`); line-level reduction is still running and I will attach the reduced case.
Two traps for anyone reducing this:
- Printing only the `FunctionOpInterface` does not reparse — `error: expected comdat symbol`. Dump the whole `ModuleOp`.
- Python `subprocess` reports a segfault as returncode **-11**, not 139. A delta-debug predicate checking `== 139` silently reports "no repro".
## Notes
- **Latent, unrelated:** `LoadSelect` (`Dialect/Ops.cpp:2598`) builds an `scf::IfOp` from `SubIndexOp`'s canonicalization patterns without anything guaranteeing the `scf` dialect is loaded; when it is not, `thenBlock()` returns null and `canonicalize` segfaults. Confirmed by injecting one `scf.if` into the input, which makes the identical run pass. Masked in the full pipeline because a later pass declares `scf` as a dependent dialect and MLIR loads those upfront — so it only bites truncated pipelines.
- MFEM is the first large real C++ codebase pointed at this path. LBM/XSBench/RSBench never hit any of the above; they build `-fno-exceptions` and are far smaller.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the standalone crash_min.mlir reproducer and run `enzymexlamlir-opt --affine-cfg crash_min.mlir -o /dev/null`. Trace `MoveIfToAffine::matchAndRewrite` into `AffineApplyNormalizer::renumberOneDim`, using the dumped ModuleOp and the reduced `quadrature_interpolator::Det3D<0,0,false>` case. Done means the dangling-Value crash and the listed MFEM pass failures are resolved so the build can link and its tests can run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp
- Domain
- build-system, compilers
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100