element-hq / element-hq/synapse

Improve structured logging

Open
#8,683 0 comments 0 reactions 0 assignees View on GitHub
A-Logging T-Enhancement
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#8683](https://github.com/matrix-org/synapse/issues/8683).

---

Building on #8588 there's a desire to improve the output of Synapse structured logging.

Currently a "standard" log line would be turned into JSON (this was a curl to `/_matrix/client/r0/login` to get the login flows, but other logs look similar):

```
2020-10-29 07:36:00,762 - synapse.access.http.8080 - 315 - INFO - GET-0 - - - 8080 - {None} Processed request: 0.000sec/-0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 285B 200 "GET /_matrix/client/r0/login HTTP/1.1" "curl/7.64.1" [0 dbevts]
```

This would transform to (note that whitespace was added for visibility, normally this would be a single line with no whitespaec between JSON separators):

```json
{
"log": "- - 8080 - {None} Processed request: 0.000sec/-0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 285B 200 \"GET /_matrix/client/r0/login HTTP/1.1\" \"curl/7.64.1\" [0 dbevts]",
"namespace": "synapse.access.http.8080",
"level": "INFO",
"time": 1603971360.76,
"request": "GET-0",
"scope": null,
"server_name": "localhost:8480"
}
```

A few improvements we'll want to make:

* The log line includes a formatted message, which makes it difficult to search in a structured way.
* Currently logging a message with additional fields requires including the fields twice (using the standard logging flow and the `extra` keyword to the logger).

Contributor guide

Open the contributing guide

Research direction

The issue does not name implementation files, tests, or an entry point. Start by locating Synapse's structured logging and standard logging integration, then define the desired searchable fields and avoid duplicated field handling before identifying tests that can demonstrate the completed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.