envoyproxy / envoyproxy/envoy

No Longer Able To Duplicate The Value of the X-B3-Traceid Header using request_headers_to_add

Open
#23,972 11 comments 3 reactions 1 assignee Claimed by @wbpcode View on GitHub
area/http area/tracing bug no stalebot stale
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

**If you are reporting *any* crash or *any* potential security issue, *do not*
open an issue in this repo. Please report the issue via emailing
envoy-security@googlegroups.com where the issue will be triaged appropriately.**

*Title*: No Longer Able To Duplicate The Value of the X-B3-Traceid Header using `request_headers_to_add`

*Description*:
Under previously released versions of Envoy (<1.22) we were able to duplicate the value of the `X-B3-Traceid` header to another upstream header that we use internally. This is to support customers who still expect a custom header.

We achieved this using header formatting in the `request_headers_to_add` configuration block, like so:

```
routes:
- match:
prefix: "/"
route: { cluster: app }
request_headers_to_add:
- header:
key: Legacy-Trace-Header
value: "%REQ(x-b3-traceid)%"
append: false
```

However, since 1.22, this behaviour has changed.

Now, there are two separate outcomes depending on the presence of the `x-b3-traceid` header in the downstream request:
- If the header is not sent, the custom header is missing entirely
- If the header is sent, the header is passed to the cluster with no modification.

*Repro steps*:
1. Clone my [demonstration repo](https://github.com/jacobneiltaylor/trace-test/tree/main).
2. Run `docker compose up`
3. Load the following links up in your browser, corresponding to different Envoy versions. Notice how, on 1.22 and up, the `Example-Traceid` header is absent in the echoed response for all versions > 1.21.
- [v1.19.0](http://localhost:1190)
- [v1.20.0](http://localhost:1200)
- [v1.21.0](http://localhost:1210)
- [v1.22.0](http://localhost:1220)
- [v1.23.0](http://localhost:1230)
- [v1.24.0](http://localhost:1240)
4. Send a crafted response to `http://localhost:1210` with the header `x-b3-traceid` set to `foobar`. Notice that the `Example-Traceid` header value seen in the JSON response is overwritten by the trace ID generated by Envoy.
5. Send a crafted response to `http://localhost:1220` with the header `x-b3-traceid` set to `foobar`. Notice that the `Example-Traceid` header value seen in the JSON response is *not* overwritten by the trace ID generated by Envoy.

If this is not the expected way to achieve this, then we'll happily use an alternative configuration pattern to achieve the same result elsewhere.

>**Note**: The [Envoy_collect tool](https://github.com/envoyproxy/envoy/blob/main/tools/envoy_collect/README.md)
gathers a tarball with debug logs, config and the following admin
endpoints: /stats, /clusters and /server_info. Please note if there are
privacy concerns, sanitize the data prior to sharing the tarball/pasting.

*Admin and Stats Output*:
>Include the admin output for the following endpoints: /stats,
/clusters, /routes, /server_info. For more information, refer to the
[admin endpoint documentation.](https://www.envoyproxy.io/docs/envoy/latest/operations/admin)

>**Note**: If there are privacy concerns, sanitize the data prior to
sharing.

*Config*: [Available here](https://github.com/jacobneiltaylor/trace-test/blob/main/config/1.24.0_envoy.conf.yaml)

*Logs*:
>Include the access logs and the Envoy logs.

>**Note**: If there are privacy concerns, sanitize the data prior to
sharing.

*Call Stack*:
> If the Envoy binary is crashing, a call stack is **required**.
Please refer to the [Bazel Stack trace documentation](https://github.com/envoyproxy/envoy/tree/main/bazel#stack-trace-symbol-resolution).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.