rust-lang / rust-lang/rust

DllMain is never run when given stdcall decorated export

Open
#135,188 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-dynamic-library A-linkage C-bug O-windows T-compiler
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

I'm creating a Windows DLL in an admittedly goofy fashion: I have an executable that links a DLL and imports DllMain, using stdcall decorations (so in reality it's looking for _DllMain@12). This causes the DllMain function to not actually run when the DLL is linked. I made a dummy DllMain function that just returns false to prevent the linking program from running, making it obvious that my code is running, but the program is starting successfully. It's worth noting that I'm running into this when testing out replacing a C DLL with a new, Rust-based DLL. My C version natively exports _DllMain@12

Using the common wisdom of using #[no_mangle] on DllMain, it is failing to link with my executable because it does not export _DllMain@12. When modifying the executable to import DllMain, my own DllMain function runs, causing the program to fail to start (as intended, in this test case).

When testing with a program that instead uses LoadLibrary(), my code will run in both cases, indicating that there's different behavior for a DLL being linked and loaded programmatically. I'm not an expert on how DLLs are loaded, but it seems like there's an entrypoint function that is run when the DLL is linked, and the entrypoint will only run DllMain when its symbol is specifically DllMain instead of the also valid _DllMain@12.

I tried this code:

#[unsafe(export_name = "_DllMain@12")]
pub extern "stdcall" fn DllMain(_hinst_dll: usize, _fdw_reason: u32, _lp_reserved: usize) -> i32 {
    0
}

(crate-type is set to cdylib)

I expected to see this happen: When the DLL is linked, the program fails due to DllMain returning false

Instead, this happened: The DLL links successfully, with DllMain seemingly returning true

Meta

rustc --version --verbose:

$ rustc --version --verbose
rustc 1.83.0 (90b35a623 2024-11-26)
binary: rustc
commit-hash: 90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf
commit-date: 2024-11-26
host: x86_64-pc-windows-msvc
release: 1.83.0
LLVM version: 19.1.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided Rust cdylib reproduction on x86_64-pc-windows-msvc, focusing on the #[unsafe(export_name = "_DllMain@12")] DllMain entry point. Compare behavior when the DLL is linked against behavior through LoadLibrary(); done means the stdcall-decorated export is invoked correctly in the linked case, matching the reported expectation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.