rawLinAlgDeterminant over the FLINT integer ring segfaults and corrupts the heap whenever an entry is zero; ZZFlint should be fixed or removed

This issue was triaged from [`bugs/mike/test-flint.m2`](https://github.com/Macaulay2/M2/blob/388c1ff0ce30d83751dea7bc7eac77fdc1305dd7/bugs/mike/test-flint.m2), one of the 857 files removed from the pre-GitHub `bugs/` tree by [`d2c8d27826`](https://github.com/Macaulay2/M2/commit/d2c8d27826) and catalogued in [#36](https://github.com/Macaulay2/M2/issues/36). **The commentary below was written by Claude (Claude Opus 5, via Claude Code)**, not by @d-torrance, whose account posted it -- please weigh it accordingly.

The original file, verbatim (256 lines)

```m2

debug Core
R = rawARingZZFlint()
a = 1_R
b = 2_R
rawRing a
a+b
a-b
a^3
a = 23746237846237846237846237846237846327846178623418764
a^4
-a
a*b
b*b
b^2
M = rawMutableMatrix(R, 3, 4, true)
rawMutableMatrixFillRandomDensity(M,1.0,0);
M

N = rawMutableMatrix(R, 4, 5, true)
rawMutableMatrixFillRandomDensity(N,1.0,0);
N
rawLinAlgMult(M,N)

E = {{99, 76, 12}, {78, 28, 96}, {43, 7, 45}}
det matrix E
M = rawMutableMatrix(R, 3, 3, true)
for i from 0 to 2 do for j from 0 to 2 do M_(i,j) = (E#i#j)_R
M
rawLinAlgDeterminant M == 97284
rawLinAlgMult(M,M)

M = rawMutableMatrix(R, 1000, 1000, true);
rawMutableMatrixFillRandomDensity(M,1.0,0);
M
time rawLinAlgMult(M,M);
rawLinAlgDeterminant M

fillMatrix M
R101 = rawARingZZpFlint(101)
select(2^36+1 .. 2^36 + 500, isPrime)
P = 68719476767
R = rawARingZZpFlint P
a = 10_R
a^(P-1)
a^(P^2-1)
a^(P^10)

select(2^64-500 .. 2^64-1, isPrime
select(2^24-500 .. 2^24-1, isPrime)
last oo
P = 18446744073709551557
-- P = 18,446,744,073,709,551,557
R = rawARingZZpFlint P
char R
random R
rawRandom R
R = ZZ/32003
random R

debug Core
R = ZZp(P, "Choose"=>"FLINT")
1_R + (-1_R)
M = 342342342342353242
N = 986908456830938608
M_R
N_R
M_R * N_R == (M*N)_R
M*N

M = mutableMatrix(R,4,5)
fillMatrix M
M_(2,3)
entries M
N1 = matrix M
N2 = transpose N1
N1*N2
det oo
det(N2*N1) -- no good yet.

R1 = ZZp(P, "Choose"=>"FFPACK") -- error, as it should be
R2 = ZZp(P, "Choose"=>"ARING") -- error, as it should be
R3 = ZZp(P) -- error, as it should be
R4 = ZZp(P, "Choose"=>"FLINT")
A = random(R4^4, R4^5)
B = random(R4^5, R4^4)
C = A*B
D = B*A
rank D
det oo
B*A
det oo

R1 = ZZp 101
R2 = ZZp 101
assert(R1 === R2)
R3 = ZZp(101, "Choose"=>"ARING")
R4 = ZZp(101, "Choose"=>"ARING")
assert(R1 =!= R3)
assert(R4 === R3)

restart
debug Core
R = ZZp(101, "Choose"=>"FFPACK")
N = 10
M = mutableMatrix(R, N, 2*N)
fillMatrix M;
time rawLinAlgRank raw M
M = mutableMatrix(R, N, N)
fillMatrix M;
time rawLinAlgDeterminant raw M

R = ZZp(101, "Choose"=>"FLINT")
M = mutableMatrix(R, N, 2*N)
fillMatrix M;
time rawLinAlgRank raw M

P = 18446744073709551557
R = ZZp(P, "Choose"=>"FLINT")
M = mutableMatrix(R, N, 2*N)
fillMatrix M;
time rawLinAlgRank raw M

P = 16777213
R = ZZp(P, "Choose"=>"FFPACK")
N = 2000
M = mutableMatrix(R, N, 2*N)
fillMatrix M;
time rawLinAlgRank raw M

R = ZZp(P, "Choose"=>"FLINT")
N = 2000
M = mutableMatrix(R, N, 2*N)
fillMatrix M;
time rawLinAlgRank raw M

M = mutableMatrix(R, 5, 4)
fillMatrix M
N = mutableMatrix(R,4,5)
fillMatrix N
(matrix N) * (matrix M)
mutableMatrix oo
rawLinAlgRank raw oo
rank M
rank matrix M

TEST ///
-- test of small matrix multiplication in flint and ffpack
restart
debug Core
R1 = ZZp(101, "Choose"=>"FFPACK")
R2 = ZZp(101, "Choose"=>"FLINT")
E = {{1,4},{7,1}}
E2 = {{3,34},{17,22}}
M1 = matrix E
M2 = matrix E2
(M1*M2) ** ZZ/101
ansMult = {{71,21}, {38, 58}}

-- First do ffpack
M1 = mutableMatrix matrix(R1, E)
M2 = mutableMatrix matrix(R1, E2)
assert(ansMult == entries rawLinAlgMult(raw M1,raw M2))

-- now flint
M1 = mutableMatrix matrix(R2, E)
M2 = mutableMatrix matrix(R2, E2)
assert(ansMult == entries rawLinAlgMult(raw M1,raw M2))
assert(M1_(0,1) == 4)
///

TEST ///
-- timing for matrix mult
restart
debug Core
N = 3
R1 = ZZp(101, "Choose"=>"FFPACK")
R2 = ZZp(101, "Choose"=>"FLINT")

M1 = mutableMatrix(R1, N, 3*N)
M2 = mutableMatrix(R1, 3*N, N)
fillMatrix M1;
fillMatrix M2;
time rawLinAlgMult(raw M1, raw M2); -- not working yet...
time rawLinAlgDeterminant raw M1

M1 = mutableMatrix(R2, N, N)
M2 = mutableMatrix(R2, N, N)
fillMatrix M1;
fillMatrix M2;
time rawLinAlgMult(raw M1, raw M2); -- not working yet...
time rawLinAlgDeterminant raw M1

///
TEST ///
-- Test of determinant
debug Core
E = {{86, 13, 36, 39, 39, 88, 7, 0, 66, 86}, {23, 10, 77, 15, 25, 33, 30, 29, 45, 13}, {77, 9, 78, 34, 7, 40, 52, 82, 36, 55}, {66, 100, 92, 27, 87, 97, 32, 6, 96, 29}, {81, 79, 21, 50, 56, 80, 28, 94, 93, 60}, {22, 15, 1, 36, 35, 59, 74, 46, 86, 31}, {82, 83, 98, 40, 19, 11, 14, 78, 29, 94}, {28, 16, 43, 44, 90, 33, 71, 34, 62, 27}, {28, 49, 81, 85, 98, 31, 85, 65, 78, 70}, {1, 33, 4, 92, 17, 17, 69, 27, 40, 30}}
(det matrix E) % 101

E = {{1,4},{7,1}}
E2 = {{3,34},{17,22}}
R = ZZ/101
M = matrix(R, E)
M2 = matrix(R,E2)
det M
M*M2

R = ZZp(101, "Choose"=>"FFPACK")
M = mutableMatrix matrix(R, E)
M2 = mutableMatrix matrix(R, E2)
rawLinAlgDeterminant(raw M)
rawLinAlgMult(raw M, raw M2)

R = ZZp(101, "Choose"=>"FLINT")
M = mutableMatrix matrix(R, E)
M2 = mutableMatrix matrix(R, E2)
rawLinAlgDeterminant(raw M)
rawLinAlgMult(raw M, raw M2)

N = 4000
N = 1000
N = 2
debug Core
R0 = ZZp(101)
M = mutableMatrix(R0, N, N)
fillMatrix M;
Ma = matrix M;
time det Ma -- not working

R1 = ZZp(101, "Choose"=>"FFPACK")
M = mutableMatrix(R1, N, N)
fillMatrix M;
time rawLinAlgDeterminant(raw M);
time rawLinAlgInvert(raw M) ;
M2 = mutableMatrix(R1, N, N)
fillMatrix M2;
rawLinAlgMult(raw M, raw M2)

R2 = ZZp(101, "Choose"=>"FLINT")
M = mutableMatrix(R2, N, N)
fillMatrix M;
time rawLinAlgDeterminant(raw M);
time rawLinAlgInvert(raw M);

P = 18446744073709551557
R3 = ZZp(P, "Choose"=>"FLINT")
M = mutableMatrix(R3, N, N)
fillMatrix M;
time rawLinAlgDeterminant(raw M);
time rawLinAlgInvert(raw M) ;


///
```

### Where it stands today

`rawLinAlgDeterminant` on a matrix over the FLINT integer ring segfaults, and corrupts the heap on the way
out, whenever any entry is zero. The 2-by-2 identity matrix is enough:

```m2
i1 : debug Core

i2 : rawLinAlgDeterminant raw mutableMatrix matrix(ZZFlint, {{1,0},{0,1}})
-- SIGSEGV
-* stack trace, pid: 233862
3# 0x00007E5E73BA6DA2 in /lib/x86_64-linux-gnu/libgmp.so.10
4# __gmpn_sqrtrem in /lib/x86_64-linux-gnu/libgmp.so.10
5# __gmpz_sqrtrem in /lib/x86_64-linux-gnu/libgmp.so.10
6# fmpz_sqrtrem in /lib/x86_64-linux-gnu/libflint.so.18
7# 0x00007E5E73D4E920 in /lib/x86_64-linux-gnu/libflint.so.18
8# fmpz_mat_solve_bound in /lib/x86_64-linux-gnu/libflint.so.18
9# fmpz_mat_solve_dixon in /lib/x86_64-linux-gnu/libflint.so.18
10# fmpz_mat_det_divisor in /lib/x86_64-linux-gnu/libflint.so.18
*-
Fatal glibc error: malloc.c:2599 (sysmalloc): assertion failed: (old_top == initial_top (av) &&
old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) &&
((unsigned long) old_end & (pagesize - 1)) == 0)
```

The `glibc` line is the part I would weight most: this is not a clean null dereference but heap corruption,
so a call that does *not* crash outright may already have damaged unrelated allocations.

### The trigger is a zero entry

| matrix over `ZZFlint` | `det` over `ZZ` | `rawLinAlgDeterminant` |
| --- | --- | --- |
| `{{1,1},{1,2}}` | 1 | 1 |
| `{{0}}` | 0 | 0 |
| `{{90,11,91},{85,34,1},{69,68,80}}` | 477133 | 477133 |
| `{{1,0},{0,1}}` | 1 | **SIGSEGV** |
| `{{5,0},{0,5}}` | 25 | **SIGSEGV** |
| `{{1,2},{3,0}}` | −6 | **SIGSEGV** |
| `{{90,11,91},{85,34,0},{69,68,80}}` | 482494 | **SIGSEGV** |

The last two rows differ in one entry: changing the single `0` to a `1` turns a crash into the correct
answer. Zero-free matrices appear to be fine at every size I tried; a dense random 1000-by-1000 also
crashes, which is where I first hit this.

### This is known, but recorded nowhere findable

The tests for it are deliberately switched off. `packages/EngineTests.m2:205-207` reads

```m2
ringsPID = {
"ZZ"
--"ZZFlint"
}
```

and the corresponding block in `packages/EngineTests/LinearAlgebra.Test.FLINT.m2:200-206` is a bare `///`
string rather than `TEST ///`, so it never runs:

```m2
///
debug Core
-- Most of this code is designed for fields...
R = ZZFlint
testDeterminant R
testMult R
///
```

Both were done by [`50058c0907`](https://github.com/Macaulay2/M2/commit/50058c0907), Mike Stillman,
2014-06-09, whose message is *"Changes to tests. One test is still failing: crashing bug."* So the crash has
been known for twelve years, and the only record of it is that commit message plus two commented-out lines.
Nothing in the issue tracker mentions `ZZFlint` or `QQFlint` at all.

### Two ways to close this, and I think either is fine

**Fix it and re-enable the tests.** `testDeterminant` and `testMult` are already written for this ring and
would exercise it the moment `"ZZFlint"` goes back into `ringsPID`. Whether the fault is in M2's call into
`fmpz_mat_det`/`fmpz_mat_det_divisor` or in FLINT itself I have not established — the stack bottoms out in
GMP's `sqrtrem` beneath FLINT's Dixon solver bound computation, so a wrong dimension or an uninitialised
`fmpz_mat` handed across the boundary would fit.

**Or remove the ring.** `ZZFlint` is not in `m2/exports.m2` and has no documentation node, so it is
unreachable without `debug Core`; it exists as `ZZFlintRing` in `m2/flint.m2:6-13`. A half-present
experimental ring that corrupts the heap is arguably worse than no ring, and twelve years without work
suggests nobody is depending on it. `QQFlint` is in the same position — `EngineTests.m2:201` carries
`-- QQFlint not working yet`, and its own test block at `LinearAlgebra.Test.FLINT.m2:208` is likewise inert
and annotated `-- testRank R --FAILS`.

What I would not recommend is leaving it as it is, since the current state costs a reader the same
investigation twice.

### Not these

[#487](https://github.com/Macaulay2/M2/issues/487) was a determinant segfault, but at the M2 level over
polynomial rings; it was fixed by [#2546](https://github.com/Macaulay2/M2/pull/2546), the `Strategy =>
Dynamic` cofactor implementation, and [#3304](https://github.com/Macaulay2/M2/pull/3304) added the
regression test for it. [#1136](https://github.com/Macaulay2/M2/issues/1136) does name
`rawLinAlgDeterminant`, but over `ZZ/2[a..d]`, and reports a questionable answer rather than a crash.
Searched titles for `ZZFlint`, `QQFlint`, `flint`, `determinant`, `rawLinAlgDeterminant`, `fmpz`, `sqrtrem`,
`heap`, `malloc`, `glibc` and `linear algebra`, bodies for `ZZFlint`, and comments for `ZZFlint`, `QQFlint`,
`rawLinAlgDeterminant` and `fmpz_mat_det_divisor`.

_{`open` · disposition `issue` · source of truth: [`bug-triage/catalog.tsv`](https://github.com/d-torrance/M2/blob/bug-triage/bug-triage/catalog.tsv)}

Reproduce the failure with rawLinAlgDeterminant on the 2-by-2 ZZFlint identity matrix, then inspect m2/flint.m2:6-13 and the determinant path involving FLINT. Read packages/EngineTests.m2:205-207 and packages/EngineTests/LinearAlgebra.Test.FLINT.m2:200-206 to understand the disabled coverage. Done means either fixing the crash and re-enabling the tests, or removing the unusable ring.

Written by the indexing model from the issue text.