open-telemetry / open-telemetry/opentelemetry-cpp-contrib
Traceparent is incorrect when the incoming request contains large headers
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 153
- Forks
- 184
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 26
Description
Describe your environment
Docker
Nginx 1.23.1 (official image)
OpenTelemetry Webserver 1.0.3 x64 Linux
Node.js / Express app acting as a "backend" (Nginx proxy_pass requests to this backend)
Request > Nginx > Express app
Steps to reproduce
- Set up Nginx and OpenTelemetry module as per the guide on https://opentelemetry.io/blog/2022/instrument-nginx/
- Start the application
- Using Postman or your HTTP client of choice, set up a request which includes a really large header (lots of key/value header pairs). For example, paste the following into the header "Bulk Edit" feature in Postman for speed:
aaaa: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
bbbb: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
cccc: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
dddd: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
eeee: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
ffff: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
gggg: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
hhhh: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
iiii: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
jjjj: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
kkkk: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
llll: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
mmmm: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
nnnn: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
oooo: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
pppp: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
qqqq: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
What is the expected behavior?
The traceparent header logged within Nginx access logs and received by the backend application on its incoming request is valid and contains a single trace, for example 00-82c5500f40667e5500e9ae8e9711553c-992631f881f78c3b-01
What is the actual behavior?
The traceparent header contains lots of comma-separated traces, belonging to each of the internal Nginx modules, for example: 00-82c5500f40667e5500e9ae8e9711553c-992631f881f78c3b-01, 00-82c5500f40667e5500e9ae8e9711553c-6ec482f97a7806d7-01, 00-82c5500f40667e5500e9ae8e9711553c-02493c5c29e4be9e-01, 00-82c5500f40667e5500e9ae8e9711553c-585000d699b02a5e-01, 00-82c5500f40667e5500e9ae8e9711553c-d5e2b37e1d74fe9c-01
Additional context
Seems to be the exact same symptoms as the issue discussed here https://opentelemetry.io/blog/2022/debug-otel-with-otel/, but only reproducible with a large set of headers. Using a small set of headers does not trigger this bug.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with Docker, Nginx 1.23.1, OpenTelemetry Webserver 1.0.3, and the Node.js/Express backend using the large-header request described. Inspect traceparent propagation through the Nginx modules; done means the access log and backend receive one valid traceparent rather than comma-separated traces.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, express, nginx, node.js
- Domain
- backend, devops, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100