element-hq / element-hq/synapse

Make log correlation more configurable (support analysis services, e.g. Google, New Relic)

Open
#14,413 0 comments 0 reactions 0 assignees View on GitHub
A-Logging O-Occasional S-Minor T-Enhancement
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#14413](https://github.com/matrix-org/synapse/issues/14413).

---

**Description:**

Thanks everyone who works on Synapse, it's really great.

The recent PR https://github.com/matrix-org/synapse/pull/13801 added the ability to specify the header from which to pull a request ID. This is a great step toward log correlation, but it more-or-less requires manual action to correlate.

I am using Google's Cloud Logging (aka Stackdriver) and logs may be correlated automatically in their tooling if the structured logs contain a `trace` key with a value [following a particular convention](https://cloud.google.com/trace/docs/trace-log-integration#associating). Something similar exists for New Relic, see discussion at https://github.com/Seldaek/monolog/issues/1735).

It would be awesome if Synapse could support this. I think it could be as simple as:

* Make the name of the request ID key, currently fixed to `request`, configurable. (E.g., I would set this to `trace`).
* Drop the auto-prefixing of the request with the method name. The method is already included in the log (`method`).

In my case, I am currently setting the trace-id with a little bit of lua magic with an openresty reverse proxy, e.g.:

```
server {
location @default {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto 'https';
set_by_lua $trace_id 'return string.sub(ngx.req.get_headers()["traceparent"], 4, 35)';
proxy_set_header trace-id $trace_id;
set $upstream {{ .Env.UPSTREAM }};
proxy_pass http://$upstream:{{ default .Env.UPSTREAM_PORT "8008" }};
}
}
```

And if this were supported, I would simply update this to include the additional data Google is looking for, e.g. the project ID. Synapse would then be responsible for hairpinning that back to me.

Alternatively, I suppose this could be viewed as more an extension of the opentracing implementation, and Synapse could read out the Trace ID from the `traceparent` header and include it somehow in the logs. However, that wouldn't address the need for making the log value configurable (e.g., prefixing it with additional cloud provider namespacing.)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.