drwrap handling of stack unwinding via SEH/longjmp

_From [bruen...@google.com](https://code.google.com/u/109494838902877177630/) on February 17, 2012 21:24:20_

drwrap handling of stack unwinding via SEH/longjmp

tailcalls are easy to handle as we see the post eventually. an unwind that bypasses all posts, though, and that may then re-grow the stack back to the same point and make esp comparisons moot, is problematic.

currently, only in post do we try to run all the post-hooks we missed. but it gets much more complex than that: on an exception, by the time we notice anything, let's say get back to a pre hook, esp can now be lower than a wrapped esp we stored. so how do we know whether to call? can of course intercept NtContinue, but what about longjmp or other arbitrary xsp change?

in DrMem, I was only hooking alloc routines, and on exception I assumed it
would travel out of the entire nested alloc stack and I reset the whole
thing to level 0. but can't assume that for arbitrary wrapping.

xref issue #120/PR 408545: handle exception in alloc routine: adjust state on unwind

first attempt:
try to handle SEH unwind and longjmp
- when notice post-calls that were bypassed abnormally,
call the post-call hook but pass NULL as wrapcxt
so user knows it's unusual
- on NtContinue, call all post-calls bypassed by target xsp,
and verify the rest by looking at retaddr to handle the case
of unwind and then re-extend stack (not bulletproof b/c of
stale retaddrs)
- after exception, verify all stored frames
- added a test that uses longjmp so cross-platform

later:
improve SEH unwind handling
- handle SEH return to same xsp
- support the user passing a flag that says to unwind
all levels when an exception occurs, instead of waiting
and trying to detect where the app unwind ends up.
this is the only way I could get DrMem and DrHeap
to work correctly: the heuristics ended up being
insufficient, the more tests I ran.

still not perfect. leaving this case open. intercepting longjmp in libc
would be a good next step though of course easy to hand-roll.

_Original issue: http://code.google.com/p/dynamorio/issues/detail?id=685_