Vector35 / Vector35/debugger

Better step return strategy for x86 binaries in native windows debug adapter

Open
#977 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

DbgEng Adapter enhancement functionality Impact: Medium
Dominant language
C++
Stars
331
Forks
33
PR merge metrics
No merged PRs in 30d

Description

Right now the native windows debug adapter uses the same strategy to do step return, i.e., placing a breakpoint at the return address and then resumes the target. While this works fine, the stack frame information returned by StackWalk function is often times inaccurate, causing the target to behave unexpectedly in various cases, e.g., in #421

There are a couple of ways to deal with. To start with, we can use binja's analysis to find return instructions and put a breakpoint on all of them. However, this does not work well for recursion. Similarly, we can consider breakpoint on ret, then continue until the stack pointer is larger, though it might break in case of tailcalls. A different way to leverage binja's analysis is since binja knows the stack pointer offset from the entry (return address) at every point in the function via LLIL stack pointer, we can find the return address on the stack and then put a breakpoint there

For context, x64dbg does step over until a return instruction is encountered

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the native Windows debug adapter's current step-return implementation and its use of StackWalk. Read the x86 step-return behavior and the Binja LLIL stack-pointer information, then compare the proposed approaches, including recursion and tailcalls. Done means a reliable return strategy that avoids the inaccurate stack-frame behavior described here.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
devtools, operating-systems, reverse-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.