SortStrategy is documented as unimplemented but is live and unvalidated: two of its values segfault and one silently returns a non-minimal resolution

Nobody has claimed this yet.

This issue was triaged from [`bugs/mike/5-res-options.m2`](https://github.com/Macaulay2/M2/blob/388c1ff0ce30d83751dea7bc7eac77fdc1305dd7/bugs/mike/5-res-options.m2), one of the 857 files removed from the pre-GitHub `bugs/` tree by [`d2c8d27826`](https://github.com/Macaulay2/M2/commit/d2c8d27826) and catalogued in [#36](https://github.com/Macaulay2/M2/issues/36). **The commentary below was written by Claude (Claude Opus 5, via Claude Code)**, not by @d-torrance, whose account posted it -- please weigh it accordingly.

The original file, verbatim (103 lines)

```m2
-- Determine the options to resolution

R = ZZ/31991 [a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, MonomialSize=>8]
I = ideal" pr-os+mt,
jr-is+gt,
pq-ns+lt,
oq-nr+kt,
mq-lr+ks,
jq-hs+ft,
iq-hr+et,
gq-fr+es,
dq-cr+bs-at,
jo-ip+dt,
mn-lo+kp,
jn-hp+ct,
in-ho+bt,
gn-fo+ep+at,
dn-co+bp,
jm-gp+ds,
im-go+dr,
hm-fo+ep+cr-bs+at,
jl-fp+cs,
il-fo+cr+at,
hl-fn+cq,
gl-fm+as,
dl-cm+ap,
jk-ep+bs-at,
ik-eo+br,
hk-en+bq,
gk-em+ar,
fk-el+aq,
dk-bm+ao,
ck-bl+an,
gh-fi+ej,
dh-ci+bj,
df-cg+aj,
de-bg+ai,
ce-bf+ah"
GEO = 2^16
AUTO1 = 2^14
AUTO2 = 2^15
LEVEL = 2^13
LEVELSTRIP = 2^13 + 2^18 -- not a minimal resolution... but generally fast to compute

DESCENDING = 8
REVERSE = 16
DEGREE = 32

COMPARE'LEX = 0
COMPARE'LEX'EXTENDED = 1
COMPARE'LEX'EXTENDED2 = 2
COMPARE'ORDER = 3
COMPARE'MONORDER = 4

-- multiply by these:
SKELETON = 1
REDUCTIONS = 64

-- skeleton sort: e.g. SKELETON*(COMPARE'LEX + DESCENDING + REVERSE + DEGREE)
-- reduction sort (always by increasing degree)

-- order is 6 bits:
-- first 3 bits give kind of sort (COMPARE'LEX, ...)
-- 2^3, 4th bit: DESCENDING
-- 2^4, 5th bit: REVERSE
-- 2^5, 6th bit: DEGREE

I = ideal flatten entries gens I;
-- time C = res(I, Strategy=>0, SortStrategy=>AUTO2 + AUTO1); BUG!!
time C = res(I, Strategy=>0, SortStrategy=>GEO+LEVEL) -- this one isn't minimal... why not? is this a BUG?

I = ideal flatten entries gens I;
time C = res(I, Strategy=>0, SortStrategy=>GEO + AUTO2 + AUTO1);
I = ideal flatten entries gens I;
time C = res(I, Strategy=>0, SortStrategy=>GEO+LEVELSTRIP) -- this one isn't minimal... why not? is this a BUG?

gbTrace = 3
I = ideal flatten entries gens I;
time C = res(I, Strategy=>0, SortStrategy=>GEO + AUTO2 + AUTO1 + SKELETON*(COMPARE'LEX + DESCENDING + REVERSE + DEGREE));
time C = res(I, Strategy=>0, SortStrategy=>GEO + AUTO2 + AUTO1 + REDUCTIONS*(COMPARE'LEX + DESCENDING + REVERSE + DEGREE));
time C = res(I, Strategy=>0, SortStrategy=>GEO + REDUCTIONS*(COMPARE'ORDER + DESCENDING + REVERSE));
time C = res(I, Strategy=>0, SortStrategy=>GEO + REDUCTIONS*(COMPARE'ORDER + DESCENDING));
time C = res(I, Strategy=>0, SortStrategy=>GEO + COMPARE'ORDER + REVERSE + REDUCTIONS*(COMPARE'ORDER + REVERSE));
time C = res(I, Strategy=>0, SortStrategy=>GEO + REDUCTIONS*(COMPARE'MONORDER));
gbTrace=0

time C = res(I, Strategy=>0)
status(C,Monomials=>true,TotalPairs=>false)
betti C === new BettiTally from {
(0,{0}) => 1,
(1,{2}) => 35,
(2,{3}) => 140,
(3,{4}) => 189,
(3,{5}) => 112,
(4,{6}) => 735,
(5,{7}) => 1080,
(6,{8}) => 735,
(7,{9}) => 112,
(7,{10}) => 189,
(8,{11}) => 140,
(9,{12}) => 35,
(10,{14}) => 1
}
```

### Where it stands today

`SortStrategy` is documented as *"Not implemented yet."* but the engine reads it, validates nothing, and
two of its bit values fail — one by crashing, one by silently returning a resolution that is not minimal.

Six variables are enough:

```m2
i1 : R = ZZ/31991[a..f];

i2 : I = ideal(a*d-b*c, a*e-b*d, b*e-c*d, a*f-c*e, b*f-d*e, c*f-e^2);

i3 : betti res(I, Strategy => 0)

0 1 2 3 4
o3 = total: 1 6 13 12 4
0: 1 . . . .
1: . 6 4 . .
2: . . 9 12 4

i4 : betti res(I, Strategy => 0, SortStrategy => 32768)
-- SIGSEGV
```

Sweeping the documented bit positions, each in a fresh process, against that baseline:

| `SortStrategy` | flag | `res(I, Strategy => 0, SortStrategy => ·)` |
| --- | --- | --- |
| `0` | — | minimal |
| `8192` | `1 << 13`, level-by-level | **not minimal** |
| `16384` | `1 << 14`, `auto_reduce = 1` | minimal |
| `32768` | `1 << 15`, `auto_reduce = 2` | **SIGSEGV** |
| `49152` | `3 << 14`, `auto_reduce = 3` | **SIGSEGV** |
| `65536` | `1 << 16`, heap-based reduction | minimal |
| `73728` | `1 << 16 \| 1 << 13` | **not minimal** |
| `131072` | `1 << 17`, by slanted degree | minimal |

### The crash

Whenever `auto_reduce >= 2`:

```
-* stack trace, pid: 40638
3# res2_poly::remove(res2term*&) const at resolutions/res-a0-poly.cpp:94
4# res2_comp::handle_pair(res2_pair*) at resolutions/res-a0.cpp:1708
5# res2_comp::do_pairs(int, int) at resolutions/res-a0.cpp:140
6# res2_comp::start_computation() at resolutions/res-a0.cpp:387
7# rawStartComputation at interface/groebner.cpp:272
*-
```

`auto_reduce` is set at `res-a0.cpp:506` from `(SortStrategy & FLAGS_AUTO) >> SHIFT_AUTO`, so the values
reaching it are `0`, `1`, `2`, `3`. Only `0` and `1` survive.

### The silently non-minimal results

`res` over a field is supposed to return a minimal free resolution, and `SortStrategy` is documented as
choosing "the strategy to be used for sorting S-pairs" — sorting S-pairs cannot change the answer. But
bit 13 does, and it does so without a warning. On the 20-variable ideal in this file the Betti total goes
from 3504 to 7770:

```
SortStrategy => 0 SortStrategy => 65536 + 8192
0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
total: 1 35 140 301 735 1080 735 301 140 35 1 total: 1 35 220 715 1449 1951 1778 1086 427 98 10
```

Note that bit 13 is not a sorting choice at all — it selects level-by-level computation, and bit 17
selects degree-by-degree. Several algorithm switches are being carried through an option whose documented
job is sorting.

The one non-minimal result that is *not* surprising is `2^13 + 2^18`, which the author's own comment in
this file marks "not a minimal resolution... but generally fast to compute". That value sets
`do_by_level = 2` at `res-a0.cpp:507`, a deliberate strip optimization. Bit 13 alone sets
`do_by_level = 1` and carries no such caveat.

### Not confined to `OldChainComplexes`

`freeResolution` in `Complexes` takes the same option and reaches the same code:

```m2
i1 : needsPackage "Complexes";

i2 : betti freeResolution(I, Strategy => 0, SortStrategy => 8192) -- not minimal
i3 : betti freeResolution(I, Strategy => 0, SortStrategy => 32768) -- SIGSEGV
```

### Two smaller things in the same code

The `auto_reduce` path writes debug tracing to stdout with `gbTrace` unset:

```
b2cf<7>-b2cf<6>+ac2f<4>-abdf<5>ad2e<9>-ad2e<1>...auto reduction:
by coeff = 1
result =
```

And the documentation node for `[resolution, SortStrategy]` in
`packages/OldChainComplexes/docs/doc10.m2:147-152` ends with "Not implemented yet.", while
`res-a0.cpp:500-507` unpacks six fields from it. `Complexes` calls it "an internal option" instead, which
is accurate but does not say that most of its values are untested.

The narrowest fix is probably to validate the option and reject the values that do not work, rather than
to repair `auto_reduce = 2`; nothing in the tree passes a nonzero `SortStrategy`, so nothing depends on
those values today.

### Searches this rests on

Titles and bodies for `SortStrategy`, `auto_reduce`, `res2_poly`, `res-a0`, `handle_pair`, `not minimal`
and `non-minimal`; comments via `gh search issues` for `SortStrategy`, `res2_poly`, `res2_comp` and
`handle_pair`. The near misses, so they are not re-derived:
[#1578](https://github.com/Macaulay2/M2/issues/1578) and
[#4143](https://github.com/Macaulay2/M2/issues/4143) both name `auto_reduce`, but that is
`gbA::auto_reduce_by` in `gb-default.cpp` — the Gröbner engine, a different function, and #4143 is about
parallel calls; [#814](https://github.com/Macaulay2/M2/issues/814) and
[#907](https://github.com/Macaulay2/M2/issues/907) list the option name only as `help` output;
[#4394](https://github.com/Macaulay2/M2/issues/4394) touches `res-a0` but is about the slab allocator.

_{`open` · disposition `issue` · source of truth: [`bug-triage/catalog.tsv`](https://github.com/d-torrance/M2/blob/bug-triage/bug-triage/catalog.tsv)}

Reproduce the six-variable example with the documented SortStrategy values, then inspect res-a0.cpp:500-507 and the crash path at resolutions/res-a0-poly.cpp:94. Compare the [resolution, SortStrategy] documentation in packages/OldChainComplexes/docs/doc10.m2:147-152 with the flags actually unpacked. Done should include behavior that no longer crashes or silently produces a non-minimal resolution, with the option documentation matching reality.

Written by the indexing model from the issue text.