retaddr write monitoring to catch inter-frame stack buffer overflows

_From [derek.br...@gmail.com](https://code.google.com/u/117968039472581148324/) on December 10, 2010 17:57:38_

PR 406731

this is an idea of mine that AFAIK no other tool, whether for security (b/c too
heavyweight for them) or for memory debugging, has attempted:

I could mark retaddr as special and detect writes to it. as long as
paying cost for mem monitoring doesn't cost extra since can fit into the
"bitlevel expansion" marker and use subtable info to distinguish "bitlevel"
from "retaddr".

if see "call" or certain "push; jmp" sequences, mark stack slot as retaddr.

"call; pop" idiom nicely handled since pop will mark as unaddressable.

if see "ret" don't need to do anything since retaddr slot will be marked
unaddressable already; could try to enforce that it uses a "retaddr" stack
slot but probably too many false positives there

there are sure to be corner case issues but I think it has fewer than
shadow stack and certainly than code-cache-retaddr. if stack pointer ever
goes past the dword marked "retaddr", Dr. Memory changes it to be marked
"unaddressable", so say on a longjmp I don't have to do any extra work.
some apps do write to retaddr deliberately (COM+, e.g.): would have to
exempt or something. some apps use "ret" for other purposes, but there can
just ignore if TOS isn't marked "retaddr", if only goal is to find writes
to the "retaddr" dword.

of course I won't catch intra-frame overflows that clobber func ptr, so not
catching all stack overflows

heavy overhead so not a production-time detector, but when used during
testing with a fuzzer can detect stack buffer overflows (at least
cross-stack-frame ones)

could perhaps be part of a paper if we have a few more novel extensions

_Original issue: http://code.google.com/p/drmemory/issues/detail?id=105_