Macaulay2 / Macaulay2/M2

Default-constructed FLINT DMat specializations are uninitialized and corrupt the heap on destruction

Open
#4,720 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Engine Linear Algebra
Dominant language
Macaulay2
Stars
435
Forks
297
Avg merge
4d 20h
Merged PRs (30d)
11

Description

Reported by Claude Opus 5, running as Claude Code on behalf of @d-torrance. The analysis and reproduction above are AI-generated; the abort was observed in a local build, but please treat the suggested fix as unverified.


Summary

The five FLINT-backed DMat specializations have a default constructor that
never initializes the underlying flint matrix, while their destructor clears it
unconditionally. Default-constructing one of these is undefined behavior:
numRows() and numColumns() read uninitialized memory, and destruction
corrupts the heap. The two GF specializations additionally dereference the
null coefficient-ring pointer, because their destructor reaches through
ring() to obtain the flint context.

The generic DMat template is not affected — its default constructor sets
mNumRows, mNumColumns and mArray, and its destructor is guarded by the
length being zero.

Affected code
File Default constructor Destructor
Macaulay2/e/basic-mutable-matrices/dmat-zz-flint.hpp 37 48
Macaulay2/e/basic-mutable-matrices/dmat-qq-flint.hpp 37 49
Macaulay2/e/basic-mutable-matrices/dmat-zzp-flint.hpp 34 45
Macaulay2/e/basic-mutable-matrices/dmat-gf-flint.hpp 34 45
Macaulay2/e/basic-mutable-matrices/dmat-gf-flint-big.hpp 33 44

Each is the same shape. From dmat-zzp-flint.hpp:

  DMat() : mRing(0) {}                          // line 34: mArray untouched
  ...
  ~DMat() { nmod_mat_clear(mArray); }           // line 45: cleared regardless

and the GF variants also read the null ring while destructing:

  ~DMat() { fq_zech_mat_clear(mArray, ring().flintContext()); }

Compare the generic template, dmat.hpp:31:

  DMat() : mRing(nullptr), mNumRows(0), mNumColumns(0), mArray(nullptr) {}
Reproducing
DMat<M2::ARingZZpFlint> empty;
std::cout << empty.numRows() << "\n";   // uninitialized
```                                     // destructor corrupts the heap

Observed while adding `unit-tests/DMatTest.cpp`, whose typed suite covers all
sixteen `DMat` coefficient rings. The first run reported implausible dimensions
and then aborted:

Expected equality of these values:
empty.numRows()
Which is: 140727511877712
0
double free or corruption (out)


### Expected

A default-constructed `DMat` should be a valid empty matrix for every
coefficient ring, as it is for the generic template: zero rows, zero columns,
and safe to destroy. Fixing it presumably means initializing the flint matrix
to 0x0 in the default constructor, and — for the `GF` specializations — either
storing the flint context or guarding the destructor on a null `mRing`.

### Current test status

`unit-tests/DMatTest.cpp` has `DISABLED_defaultConstruction`, which asserts the
intended behavior and is skipped for the five affected rings. Fixing this
should let it be re-enabled as-is.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the default constructors and destructors in dmat-zz-flint.hpp, dmat-qq-flint.hpp, dmat-zzp-flint.hpp, dmat-gf-flint.hpp, and dmat-gf-flint-big.hpp, comparing them with dmat.hpp. Re-enable DISABLED_defaultConstruction in unit-tests/DMatTest.cpp and run the DMat tests; done means every specialization reports zero dimensions and can be safely destroyed.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.