open-telemetry / open-telemetry/opentelemetry-cpp

[TEST] The gRPC functional test sometimes fails the `cert-not-found` case

Open
#4,273 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C++
Stars
1.4k
Forks
632
Avg merge
1d 13h
Merged PRs (30d)
75

Description

The grpc functional tests sometimes fails the cert-not-found case in CI with double free or corruption (fasttop). It has appeared several times with the CMake clang 18 (maintainer mode, async) job.

This needs to be investigated and patched.

###############################################################
Starting otelcol --config otel-config-https-mtls.yaml
###############################################################

a4f0912591a4c25f4b8677d1d0cf4ecc450bc6a6f706c2d31700f12a45db2c40
=====================================================================
Running test basic on insecure http://localhost:4317 with server https
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1784659071.997424   21422 ssl_transport_security.cc:2160] Handshake failed with error SSL_ERROR_SSL: error:0A000086:SSL routines::certificate verify failed: unable to get local issuer certificate
I0000 00:00:1784659072.007723   21420 ssl_transport_security.cc:2160] Handshake failed with error SSL_ERROR_SSL: error:0A000086:SSL routines::certificate verify failed: unable to get local issuer certificate
 - [E] [OTLP TRACE GRPC Exporter] Export() failed with status_code: "DEADLINE_EXCEEDED" error_message: "Deadline Exceeded"
 - [D] [OTLP GRPC Client] DEBUG: OtlpGrpcClient start to shutdown
TEST basic-insecure-https: PASSED
=====================================================================
Running test cert-not-found on insecure http://localhost:4317 with server https
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1784659072.909412   21433 ssl_transport_security.cc:2160] Handshake failed with error SSL_ERROR_SSL: error:0A000086:SSL routines::certificate verify failed: unable to get local issuer certificate
I0000 00:00:1784659072.916668   21432 ssl_transport_security.cc:2160] Handshake failed with error SSL_ERROR_SSL: error:0A000086:SSL routines::certificate verify failed: unable to get local issuer certificate
double free or corruption (fasttop)
./run_test_mode.sh: line 46: 21428 Aborted                 (core dumped) "${TEST_BIN_DIR}/${TEST_EXECUTABLE}" --debug --mode ${SERVER_MODE} --cert-dir ${CERT_DIR} --endpoint ${TEST_ENDPOINT} ${T}
Error: Process completed with exit code 134.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The failure is in the gRPC functional test's cert-not-found case, invoked by run_test_mode.sh at line 46, in the CMake clang 18 maintainer-mode async CI job. Start by reproducing that case with otel-config-https-mtls.yaml and compare it with basic-insecure-https. Done means the intermittent abort is eliminated and the functional test passes repeatedly in the affected CI configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, grpc
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.