element-hq / element-hq/synapse
Improve structured logging
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#8683](https://github.com/matrix-org/synapse/issues/8683).
---
Building on #8588 there's a desire to improve the output of Synapse structured logging.
Currently a "standard" log line would be turned into JSON (this was a curl to `/_matrix/client/r0/login` to get the login flows, but other logs look similar):
```
2020-10-29 07:36:00,762 - synapse.access.http.8080 - 315 - INFO - GET-0 - - - 8080 - {None} Processed request: 0.000sec/-0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 285B 200 "GET /_matrix/client/r0/login HTTP/1.1" "curl/7.64.1" [0 dbevts]
```
This would transform to (note that whitespace was added for visibility, normally this would be a single line with no whitespaec between JSON separators):
```json
{
"log": "- - 8080 - {None} Processed request: 0.000sec/-0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 285B 200 \"GET /_matrix/client/r0/login HTTP/1.1\" \"curl/7.64.1\" [0 dbevts]",
"namespace": "synapse.access.http.8080",
"level": "INFO",
"time": 1603971360.76,
"request": "GET-0",
"scope": null,
"server_name": "localhost:8480"
}
```
A few improvements we'll want to make:
* The log line includes a formatted message, which makes it difficult to search in a structured way.
* Currently logging a message with additional fields requires including the fields twice (using the standard logging flow and the `extra` keyword to the logger).
Contributor guide
Research direction
The issue does not name implementation files, tests, or an entry point. Start by locating Synapse's structured logging and standard logging integration, then define the desired searchable fields and avoid duplicated field handling before identifying tests that can demonstrate the completed behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100