lowRISC / lowRISC/opentitan-provisioning

Windows ATE client crashes (0xC0000005) under mTLS at -O2 - GCC/MinGW32 inlining miscompile in gRPC's CoreConfiguration::Builder/ServiceConfigParser registration

Open
#326 0 comments 0 reactions 1 assignee Claimed by @willyzha View on GitHub
Dominant language
Go
Stars
16
Forks
13
Avg merge
2d 3h
Merged PRs (30d)
18

Description

## Summary

When mTLS is enabled (`--enable_mtls=true`), the Windows ATE client (`ate_main.exe`, and

`ate.dll` as used by `secigen.exe`) crashes immediately inside `AteClient::Create`, before any

network I/O, with an access violation (`0xC0000005` / exit code `-1073741819`). The insecure

(non-mTLS) path works fine. Reproduced on a downstream fork; root-caused via gdb + disassembly.

## Environment

- Windows client built with the 32-bit MXE MinGW32 GCC toolchain (`i686-w64-mingw32`, via

`lowRISC/crt`'s `mxe-binaries-win32.tar.xz`)

- gRPC 1.68.0 (`com_github_grpc_grpc`)

- Only reproduces at an **optimized** build level (`-c opt` / `-O2`); not reproducible at

`-O0`/fastbuild.

## Root cause

A **GCC codegen bug in the 32-bit MinGW32 cross-compiler**, triggered by function inlining at

`-O2` — not a bug in the ATE client or the mTLS logic itself.

The crash is inside gRPC's own channel-init/filter-registration code:

grpc_channel_create -> GcpAuthenticationServiceConfigParser::Register

-> grpc_core::ServiceConfigParser::Builder::RegisterParser

faulting on:

mov (%edx),%ecx <-- FAULT, edx = garbage (0x0 or 0x1)

This walks a `std::vector>` (`registered_parsers_`) that should be

empty at this point, but whose begin/end pointers are garbage instead of null — i.e. it's read

before being properly constructed.

Disassembling the crashing build side-by-side against a build compiled with `-fno-inline` on

gRPC shows the mechanism. Crashing build — `std::vector::begin()`/`end()` inlined directly into

`RegisterParser` as raw field loads:

mov (%ecx),%eax ; begin() -- inlined

mov 0x4(%ecx),%edx ; end() -- inlined

cmp %edx,%eax

je ; not taken -- looks "non-empty" when it should be empty

...

mov (%edx),%ecx <-- FAULT

With `-fno-inline`, the same accessors stay real out-of-line calls:

call vector<...>::begin()

call vector<...>::end()

At `-O2`, GCC inlines the vector's construction together with the code that reads it into one

large merged function; most likely interacting with 32-bit MinGW's setjmp/longjmp-style

exception bookkeeping (visible in every one of these frames), the read appears to get scheduled

ahead of the write that zero-initializes the vector. We also double-checked the standalone

`CoreConfiguration::Builder::Builder()` constructor in isolation - it correctly zero-initializes

every member - so this is not a simple "missing constructor store"; the bug only manifests once

the constructor and `RegisterParser` are inlined together into one combined function body.

## Fix that works

build [--per_file_copt=external/grpc.*@-fno-inline](mailto:--per_file_copt=external/grpc.*@-fno-inline)

Scoped to gRPC's own sources. Runtime-verified: the mTLS handshake proceeds cleanly through

`AteClient::Create -> grpc::CreateChannel -> InitSession`. `-O0` on all of gRPC also avoids it

(bigger perf cost, more of a blunt workaround).

## Ruled out first (in case useful to others)

- `-mstackrealign` — no effect

- `-fno-strict-aliasing` — no effect

- `-fno-lifetime-dse` — no effect

- `-O1` — byte-identical output to the crashing `-O2` default

- `-fno-tree-sra -fno-ipa-sra` scoped to just `core_configuration.cc` - still crashed at

runtime despite the standalone constructor's disassembly looking correct (a reminder that

disassembly of that one function alone isn't reliable evidence here)

## Reproduction

Build the Windows ATE client at an optimized level with mTLS enabled and run it with

`--enable_mtls=true` plus valid `--client_key`/`--client_cert`/`--ca_root_certs` flags against

any gRPC target — it crashes in `AteClient::Create` before any connection attempt, so the

target doesn't even need to be reachable.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.