open-telemetry / open-telemetry/opentelemetry-cpp
[BUG] func_otlp_grpc segfaults during a TLS case after ctest reports everything passing, failing the job
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.4k
- Forks
- 632
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 75
Description
Describe your environment
GitHub hosted Linux runners, main at 1c2b007a. Seen on two different maintainer mode jobs:
CMake gcc 14 (maintainer mode, async)CMake clang 18 (maintainer mode, yaml config)
Steps to reproduce
I do not have a local reproduction. It shows up on CI, and both sightings below are on pull requests that do not touch the OTLP gRPC exporter.
What is the expected behavior?
ci/do_ci.sh reports the job's result from the tests. When ctest reports every test passing, the job passes.
What is the actual behavior?
The job fails with exit code 139 after ctest has already reported everything green. The crash is in the functional gRPC binary that runs after ctest, on a TLS case:
100% tests passed, 0 tests failed out of 1593
...
Running test cert-unreadable on secure https://localhost:4317 with server https
I0000 ssl_transport_security.cc:2160] Handshake failed with error SSL_ERROR_SSL: error:0A
./run_test_mode.sh: line 67: 34560 Segmentation fault (core dumped) "${TEST_BIN_DIR}/${TEST_EXECUTABLE}" --debug --m
##[error]Process completed with exit code 139
Two sightings, two days apart, two compilers:
| PR | job | run |
|---|---|---|
| #4453 | CMake gcc 14 (maintainer mode, async) |
97488993921 |
| #4337 | CMake clang 18 (maintainer mode, yaml config) |
97837662928 |
Additional context
Neither pull request can be the cause, and that is checkable rather than argued. functional/otlp/CMakeLists.txt links the crashing binary against two targets:
target_link_libraries(func_otlp_grpc ${CMAKE_THREAD_LIBS_INIT}
opentelemetry_trace opentelemetry_exporter_otlp_grpc)
#4337 changes only exporters/elasticsearch/, and grep -rn -i elasticsearch functional/ returns nothing. #4453 changes only exporters/otlp/src/otlp_http_client.cc and its test. Neither is in that binary's link closure.
For #4453 there is a second measurement: the head that passed this job and the head that failed it differ only in comments, 0 non-comment changed lines. One passed, one failed.
It is not constant. CMake clang 18 (maintainer mode, yaml config) was green on 11 of the last 12 main commits, so this reads as a race in the functional test or in the gRPC TLS teardown rather than something that reproduces on demand.
What it costs today is that a red job on a pull request has to be opened and read before it can be dismissed, and the log looks alarming because ctest is green immediately above it.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with functional/otlp/CMakeLists.txt and the ci/do_ci.sh and run_test_mode.sh paths that run func_otlp_grpc after ctest. Investigate the intermittent cert-unreadable TLS case and its teardown behavior using the cited CI runs and test output. Done means the TLS functional test no longer segfaults or leaves the job result misleading after ctest passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp, github-actions, grpc
- Domain
- ci-cd, security, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100