microsoft / microsoft/Detours

Returns ERROR_INVALID_BLOCK on large functions with __chkstk on it.

Open
#254 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C++
Stars
6.4k
Forks
1.2k
PR merge metrics
No merged PRs in 30d

Description

Latest version as for today(Version 4.0.1).

Trying to hook to a large function, looking at the disassembly:

     9: void function()
    10: {
00007FF6E296E010 B8 C8 6C 00 00      mov         eax,6CC8h  
00007FF6E296E015 E8 A3 3B FE FF       call        __chkstk (07FF6E2951BBDh)  
00007FF6E296E01A 48 2B E0                sub         rsp,rax 

I have the same project built at home and the __chkstk is not there, deours hook without issues. At another PC with the same compiler, it builds adding that instruction and detour fails to hook.

I assume the block is too small thanks to that call to __chkstk. I think it should be supported, but still, please let me know if this is an issue or if it's expected.

Building with VS2022 target x64, compiler options: /MD /Zi /O2 /Ob1 /DNDEBUG

Thanks.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the ERROR_INVALID_BLOCK failure with Detours 4.0.1 using VS2022 x64 and the reported /MD /Zi /O2 /Ob1 /DNDEBUG options, comparing builds with and without __chkstk. Use the disassembly in the report as the initial scope; done means establishing whether large functions containing __chkstk can be hooked or documenting why the behavior is expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.