lowRISC / lowRISC/opentitan-provisioning
Windows ATE client crashes (0xC0000005) under mTLS at -O2 - GCC/MinGW32 inlining miscompile in gRPC's CoreConfiguration::Builder/ServiceConfigParser registration
- Dominant language
- Go
- Stars
- 16
- Forks
- 13
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 18
Description
## Summary
When mTLS is enabled (`--enable_mtls=true`), the Windows ATE client (`ate_main.exe`, and
`ate.dll` as used by `secigen.exe`) crashes immediately inside `AteClient::Create`, before any
network I/O, with an access violation (`0xC0000005` / exit code `-1073741819`). The insecure
(non-mTLS) path works fine. Reproduced on a downstream fork; root-caused via gdb + disassembly.
## Environment
- Windows client built with the 32-bit MXE MinGW32 GCC toolchain (`i686-w64-mingw32`, via
`lowRISC/crt`'s `mxe-binaries-win32.tar.xz`)
- gRPC 1.68.0 (`com_github_grpc_grpc`)
- Only reproduces at an **optimized** build level (`-c opt` / `-O2`); not reproducible at
`-O0`/fastbuild.
## Root cause
A **GCC codegen bug in the 32-bit MinGW32 cross-compiler**, triggered by function inlining at
`-O2` — not a bug in the ATE client or the mTLS logic itself.
The crash is inside gRPC's own channel-init/filter-registration code:
grpc_channel_create -> GcpAuthenticationServiceConfigParser::Register
-> grpc_core::ServiceConfigParser::Builder::RegisterParser
faulting on:
mov (%edx),%ecx <-- FAULT, edx = garbage (0x0 or 0x1)
This walks a `std::vector>` (`registered_parsers_`) that should be
empty at this point, but whose begin/end pointers are garbage instead of null — i.e. it's read
before being properly constructed.
Disassembling the crashing build side-by-side against a build compiled with `-fno-inline` on
gRPC shows the mechanism. Crashing build — `std::vector::begin()`/`end()` inlined directly into
`RegisterParser` as raw field loads:
mov (%ecx),%eax ; begin() -- inlined
mov 0x4(%ecx),%edx ; end() -- inlined
cmp %edx,%eax
je ; not taken -- looks "non-empty" when it should be empty
...
mov (%edx),%ecx <-- FAULT
With `-fno-inline`, the same accessors stay real out-of-line calls:
call vector<...>::begin()
call vector<...>::end()
At `-O2`, GCC inlines the vector's construction together with the code that reads it into one
large merged function; most likely interacting with 32-bit MinGW's setjmp/longjmp-style
exception bookkeeping (visible in every one of these frames), the read appears to get scheduled
ahead of the write that zero-initializes the vector. We also double-checked the standalone
`CoreConfiguration::Builder::Builder()` constructor in isolation - it correctly zero-initializes
every member - so this is not a simple "missing constructor store"; the bug only manifests once
the constructor and `RegisterParser` are inlined together into one combined function body.
## Fix that works
build [--per_file_copt=external/grpc.*@-fno-inline](mailto:--per_file_copt=external/grpc.*@-fno-inline)
Scoped to gRPC's own sources. Runtime-verified: the mTLS handshake proceeds cleanly through
`AteClient::Create -> grpc::CreateChannel -> InitSession`. `-O0` on all of gRPC also avoids it
(bigger perf cost, more of a blunt workaround).
## Ruled out first (in case useful to others)
- `-mstackrealign` — no effect
- `-fno-strict-aliasing` — no effect
- `-fno-lifetime-dse` — no effect
- `-O1` — byte-identical output to the crashing `-O2` default
- `-fno-tree-sra -fno-ipa-sra` scoped to just `core_configuration.cc` - still crashed at
runtime despite the standalone constructor's disassembly looking correct (a reminder that
disassembly of that one function alone isn't reliable evidence here)
## Reproduction
Build the Windows ATE client at an optimized level with mTLS enabled and run it with
`--enable_mtls=true` plus valid `--client_key`/`--client_cert`/`--ca_root_certs` flags against
any gRPC target — it crashes in `AteClient::Create` before any connection attempt, so the
target doesn't even need to be reachable.
Contributor guide
Assessment
This issue has not been assessed yet.