Better step return strategy for x86 binaries in native windows debug adapter
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 331
- Forks
- 33
- PR merge metrics
- No merged PRs in 30d
Description
Right now the native windows debug adapter uses the same strategy to do step return, i.e., placing a breakpoint at the return address and then resumes the target. While this works fine, the stack frame information returned by StackWalk function is often times inaccurate, causing the target to behave unexpectedly in various cases, e.g., in #421
There are a couple of ways to deal with. To start with, we can use binja's analysis to find return instructions and put a breakpoint on all of them. However, this does not work well for recursion. Similarly, we can consider breakpoint on ret, then continue until the stack pointer is larger, though it might break in case of tailcalls. A different way to leverage binja's analysis is since binja knows the stack pointer offset from the entry (return address) at every point in the function via LLIL stack pointer, we can find the return address on the stack and then put a breakpoint there
For context, x64dbg does step over until a return instruction is encountered
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the native Windows debug adapter's current step-return implementation and its use of StackWalk. Read the x86 step-return behavior and the Binja LLIL stack-pointer information, then compare the proposed approaches, including recursion and tailcalls. Done means a reliable return strategy that avoids the inaccurate stack-frame behavior described here.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- devtools, operating-systems, reverse-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100