element-hq / element-hq/synapse

Remote logging can drop log lines

Open
#16,851 1 comment 0 reactions 0 assignees View on GitHub
A-Logging O-Uncommon S-Minor T-Defect
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

### Description

Remote logging is a feature that allows for sending logging to a remote network resource, as opposed to a local file or the console. Relevant [Synapse documentation](https://element-hq.github.io/synapse/v1.99/structured_logging.html#structured-logging).

A report from @clokep [in #synapse-dev](https://matrix.to/#/!vcyiEtMVHIhWXcJAfl:sw1v.org/$QVmwPr28tPQG-SZtNAohimIw0lYzaQixrP_kv5OGQJo?via=matrix.org&via=element.io&via=envs.net).

> Remote logging is fairly broken, just an FYI. It can end up dropping log lines during (re)connection and some other situations. This might render structured logging not as useful.

@Half-Shot chimed in to state that a workaround used in the past was to instead log to disk, and then scrape the resulting files.

@clokep said:

> It was something with the way the produced was paused and started IIRC:

https://github.com/matrix-org/synapse/blob/d58dd1d1ff283d14ada805c72aa0b704087ac9b4/synapse/logging/_remote.py#L47

### Steps to reproduce

Remote logging can be configured in the logging config. See the bottom of https://element-hq.github.io/synapse/v1.99/structured_logging.html#structured-logging.

Contributor guide

Open the contributing guide

Research direction

Start with synapse/logging/_remote.py around the linked line 47, then review the remote logging configuration at the bottom of the structured logging documentation. Reproduce logging during connection and reconnection situations and determine where lines are lost. Done means remote logging preserves log lines through those situations; add or run focused coverage if the existing test layout identifies a suitable location.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.