element-hq / element-hq/synapse
Remote logging can drop log lines
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
### Description
Remote logging is a feature that allows for sending logging to a remote network resource, as opposed to a local file or the console. Relevant [Synapse documentation](https://element-hq.github.io/synapse/v1.99/structured_logging.html#structured-logging).
A report from @clokep [in #synapse-dev](https://matrix.to/#/!vcyiEtMVHIhWXcJAfl:sw1v.org/$QVmwPr28tPQG-SZtNAohimIw0lYzaQixrP_kv5OGQJo?via=matrix.org&via=element.io&via=envs.net).
> Remote logging is fairly broken, just an FYI. It can end up dropping log lines during (re)connection and some other situations. This might render structured logging not as useful.
@Half-Shot chimed in to state that a workaround used in the past was to instead log to disk, and then scrape the resulting files.
@clokep said:
> It was something with the way the produced was paused and started IIRC:
https://github.com/matrix-org/synapse/blob/d58dd1d1ff283d14ada805c72aa0b704087ac9b4/synapse/logging/_remote.py#L47
### Steps to reproduce
Remote logging can be configured in the logging config. See the bottom of https://element-hq.github.io/synapse/v1.99/structured_logging.html#structured-logging.
Contributor guide
Research direction
Start with synapse/logging/_remote.py around the linked line 47, then review the remote logging configuration at the bottom of the structured logging documentation. Reproduce logging during connection and reconnection situations and determine where lines are lost. Done means remote logging preserves log lines through those situations; add or run focused coverage if the existing test layout identifies a suitable location.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100