open-telemetry / open-telemetry/opentelemetry-cpp-contrib

Traceparent is incorrect when the incoming request contains large headers

Open
#267 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Webserver
Dominant language
Python
Stars
153
Forks
184
Avg merge
2d 9h
Merged PRs (30d)
26

Description

Describe your environment

Docker
Nginx 1.23.1 (official image)
OpenTelemetry Webserver 1.0.3 x64 Linux
Node.js / Express app acting as a "backend" (Nginx proxy_pass requests to this backend)

Request > Nginx > Express app

Steps to reproduce

  1. Set up Nginx and OpenTelemetry module as per the guide on https://opentelemetry.io/blog/2022/instrument-nginx/
  2. Start the application
  3. Using Postman or your HTTP client of choice, set up a request which includes a really large header (lots of key/value header pairs). For example, paste the following into the header "Bulk Edit" feature in Postman for speed:
aaaa: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
bbbb: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
cccc: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
dddd: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
eeee: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
ffff: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
gggg: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
hhhh: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
iiii: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
jjjj: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
kkkk: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
llll: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
mmmm: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
nnnn: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
oooo: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
pppp: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
qqqq: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

What is the expected behavior?
The traceparent header logged within Nginx access logs and received by the backend application on its incoming request is valid and contains a single trace, for example 00-82c5500f40667e5500e9ae8e9711553c-992631f881f78c3b-01

What is the actual behavior?
The traceparent header contains lots of comma-separated traces, belonging to each of the internal Nginx modules, for example: 00-82c5500f40667e5500e9ae8e9711553c-992631f881f78c3b-01, 00-82c5500f40667e5500e9ae8e9711553c-6ec482f97a7806d7-01, 00-82c5500f40667e5500e9ae8e9711553c-02493c5c29e4be9e-01, 00-82c5500f40667e5500e9ae8e9711553c-585000d699b02a5e-01, 00-82c5500f40667e5500e9ae8e9711553c-d5e2b37e1d74fe9c-01

Additional context
Seems to be the exact same symptoms as the issue discussed here https://opentelemetry.io/blog/2022/debug-otel-with-otel/, but only reproducible with a large set of headers. Using a small set of headers does not trigger this bug.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with Docker, Nginx 1.23.1, OpenTelemetry Webserver 1.0.3, and the Node.js/Express backend using the large-header request described. Inspect traceparent propagation through the Nginx modules; done means the access log and backend receive one valid traceparent rather than comma-separated traces.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, express, nginx, node.js
Domain
backend, devops, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.