open-telemetry / open-telemetry/opentelemetry-cpp

[BUG] func_otlp_grpc segfaults during a TLS case after ctest reports everything passing, failing the job

Open
#4,489 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug triage/accepted
Dominant language
C++
Stars
1.4k
Forks
632
Avg merge
1d 13h
Merged PRs (30d)
75

Description

Describe your environment

GitHub hosted Linux runners, main at 1c2b007a. Seen on two different maintainer mode jobs:

  • CMake gcc 14 (maintainer mode, async)
  • CMake clang 18 (maintainer mode, yaml config)

Steps to reproduce

I do not have a local reproduction. It shows up on CI, and both sightings below are on pull requests that do not touch the OTLP gRPC exporter.

What is the expected behavior?

ci/do_ci.sh reports the job's result from the tests. When ctest reports every test passing, the job passes.

What is the actual behavior?

The job fails with exit code 139 after ctest has already reported everything green. The crash is in the functional gRPC binary that runs after ctest, on a TLS case:

100% tests passed, 0 tests failed out of 1593
...
Running test cert-unreadable on secure https://localhost:4317 with server https
I0000 ssl_transport_security.cc:2160] Handshake failed with error SSL_ERROR_SSL: error:0A
./run_test_mode.sh: line 67: 34560 Segmentation fault (core dumped) "${TEST_BIN_DIR}/${TEST_EXECUTABLE}" --debug --m
##[error]Process completed with exit code 139

Two sightings, two days apart, two compilers:

PR job run
#4453 CMake gcc 14 (maintainer mode, async) 97488993921
#4337 CMake clang 18 (maintainer mode, yaml config) 97837662928

Additional context

Neither pull request can be the cause, and that is checkable rather than argued. functional/otlp/CMakeLists.txt links the crashing binary against two targets:

target_link_libraries(func_otlp_grpc ${CMAKE_THREAD_LIBS_INIT}
                      opentelemetry_trace opentelemetry_exporter_otlp_grpc)

#4337 changes only exporters/elasticsearch/, and grep -rn -i elasticsearch functional/ returns nothing. #4453 changes only exporters/otlp/src/otlp_http_client.cc and its test. Neither is in that binary's link closure.

For #4453 there is a second measurement: the head that passed this job and the head that failed it differ only in comments, 0 non-comment changed lines. One passed, one failed.

It is not constant. CMake clang 18 (maintainer mode, yaml config) was green on 11 of the last 12 main commits, so this reads as a race in the functional test or in the gRPC TLS teardown rather than something that reproduces on demand.

What it costs today is that a red job on a pull request has to be opened and read before it can be dismissed, and the log looks alarming because ctest is green immediately above it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with functional/otlp/CMakeLists.txt and the ci/do_ci.sh and run_test_mode.sh paths that run func_otlp_grpc after ctest. Investigate the intermittent cert-unreadable TLS case and its teardown behavior using the cited CI runs and test output. Done means the TLS functional test no longer segfaults or leaves the job result misleading after ctest passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, github-actions, grpc
Domain
ci-cd, security, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.