`stackmargin` results are misleading

(This could arguably be in the Humility repo, but I suspect that any changes we make to fix the problem will be in Hubris, so I'm filing it here; YOLO.)

tl;dr: The humility stackmargin results can lull you into a false sense of security. We should figure out how to fix this.

What stackmargin does

When a task is (re)initialized, we fill the stack memory with a recognizable pattern of bits. humility stackmargin scans the task's stack area from low addresses to high, and reports the first point where this recognizable pattern of bits is altered. This is roughly the highest watermark of used stack memory in this incarnation of the task.

What programs do

The issue is that programs don't completely write their stack frames. There are usually locals allocated in the stack that are only used on conditional paths. This is not theoretical -- we've been bitten by this in a couple of situations:

• A function in a task allocates like 2048 bytes of stack. This bumps the value of the SP register outside of the task's allocated stack region. (On ARMv7-M the hardware does not detect this. On ARMv8-M it can.)
• The function only uses locals in, say, the high 32 bytes of that region. This means there are no actual writes outside of the allocated stack region, so no memory management fault occurs.
• This situation is actually fine... until an interrupt occurs with the SP out of range. At that point, the hardware notices that it's not able to deposit the exception frame on the stack, and the task is killed.
• Because the upper bytes of the stack frame are not written, this is invisible to stackmargin.

This means that an "out of stack" condition can hide, unnoticed by either the kernel or stackmargin, for an arbitrary period of time until it happens to be noticed by an interrupt. (At that point, it remains invisible to stackmargin, since stackmargin doesn't consider crashed tasks.)

And so

So stackmargin right now is always an over-estimate. In some cases, it's an extreme over-estimate. This is bad, because we use stackmargin to check whether e.g. a compiler upgrade will cause problems by changing inlining decisions. Currently, we could observe a higher stackmargin but actually have a new stack overflow problem (if the leaf function in the deepest stack has a very large partially used frame).

We should figure out how to fix this.

I'd like to open the floor to brainstorming on this. Here are my initial thoughts.

• Hardware stack limits are only a partial solution: The ARMv8-M stack limit registers are the right way to handle this "hidden overflow" condition, but --- counter-intuitively! --- don't actually improve stackmargin output. The SP can be bumped up to the stack limit, but if the final frame is only partially written, stackmargin will still insist we have plenty of stack free. (Hardware stack limits do at least limit us to not going below 0 stackmargin.)
• Any kernel entry or interrupt already affects stack margin. Since a syscall or interrupt begins by writing the task's state to its stack, stackmargin will be able to see the deepest stack at the time of kernel entry. This implies that any method for measuring transient stack pointers that involves a kernel entry --- such as sampling SP from a timer interrupt --- will crash a task that has silently run out of stack, which is arguably good. However, note that we already take regular interrupts, and tasks have been able to overflow their stacks without taking one. We could crank up interrupt frequency to defend against this, but at the cost of CPU utilization and latency. (I don't like this class of approach.)
• Toolchain support for stack probing and the like might help. Rust has existing support for emitting stack probes as stack frames get bigger, which prevents this class of problem on virtually addressed machines. These are not enabled by default on our targets, but we could potentially fix that. We'd need to look carefully at the stack probes --- if they're written by Big Computer People who assume paged memory, they may only probe the lowest address in each page, whereas we want them to probe the lowest address, period, and not round to any page boundary (since we don't have pages). This would produce a proactive crash like a stack limit register, but would not help stackmargin. If the stack probe could be made to do a write, however... that would get us exact results!

Start with the humility stackmargin behavior described in the issue, then inspect Hubris stack initialization and the Rust target/toolchain support for stack probes. Compare the reported watermark with partially written stack frames and the ARM stack-limit discussion. Done means an agreed, implemented approach that detects or accurately reports hidden stack overflow.