Macaulay2 / Macaulay2/M2

Catch SIGSEGV on an alternate stack with sigaltstack

Open
#4,543 1 comment 0 reactions 0 assignees View on GitHub
bugs directory Interpreter
Dominant language
Macaulay2
Stars
435
Forks
297
Avg merge
4d 20h
Merged PRs (30d)
11

Description

This issue was triaged from [`bugs/dan/0.5-alternate-stack`](https://github.com/Macaulay2/M2/blob/388c1ff0ce30d83751dea7bc7eac77fdc1305dd7/bugs/dan/0.5-alternate-stack), one of the 857 files removed from the pre-GitHub `bugs/` tree by [`d2c8d27826`](https://github.com/Macaulay2/M2/commit/d2c8d27826) and catalogued in [#36](https://github.com/Macaulay2/M2/issues/36). **The commentary below was written by Claude (Claude Opus 5, via Claude Code)**, not by @d-torrance, whose account posted it -- please weigh it accordingly.

### The original file, verbatim

```text
use sigaltstack to catch SIGSEGV on an alternate stack
```

### Where it stands today

Never done: `sigaltstack`, `SA_ONSTACK` and `SIGSTKSZ` appear **nowhere** in the tree, so a `SIGSEGV`
raised by stack exhaustion still cannot be caught.

### Why this is not hypothetical

M2 has a `SIGSEGV` handler that prints a useful trace — for a Gröbner basis computation over an engine
tower ring, for instance:

```
-- SIGSEGV
3# GaussElimComputation::insert(gm_elem*) at e/gauss.cpp:50
...
```

But it goes silent in exactly the case where the crash is most likely, because the handler has no stack
to run on. Measured with the default 8192K stack, on a file containing a single long list literal:

| list length | result |
| ---: | --- |
| 150000 | parses |
| 175000 | **SIGSEGV**, exit 139, no output at all |
| 300000 at `ulimit -s 16384` | parses |

So the parser's recursion depth scales with the input, and crossing the limit produces a bare
segmentation fault with no diagnosis. Machine-generated M2 code reaches these sizes routinely — writing
`toExternalString` of a large list to a file and reading it back is an ordinary workflow.

### What an alternate stack would buy

The handler would run, so the user would at least learn where the process died instead of getting
nothing. Better still, with the crash catchable the parser could raise "expression nested too deeply"
before reaching it.

`open` · disposition `issue` · source of truth: [`bug-triage/catalog.tsv`](https://github.com/d-torrance/M2/blob/bug-triage/bug-triage/catalog.tsv)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the existing SIGSEGV handler and the parser path that overflows the stack; e/gauss.cpp:50 is an example of the diagnostic trace the handler can produce. Reproduce the long-list failure described in the issue, then verify that SIGSEGV runs on an alternate stack and emits a useful diagnosis instead of exiting silently.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
operating-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.