element-hq / element-hq/synapse

Unhandled ijson exceptions in various places causing log and sentry spam

Open
#14,320 0 comments 0 reactions 0 assignees View on GitHub
A-Logging O-Uncommon S-Tolerable T-Defect Z-Dev-Wishlist Z-Sentry
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#14320](https://github.com/matrix-org/synapse/issues/14320).

---

https://sentry.tools.element.io/organizations/element/issues/29/?project=2&query=is%3Aunresolved&statsPeriod=24h

```
2022-10-28 02:43:48,424 - twisted - 274 - ERROR - _maybe_retry_device_resync-852- - Exception ignored in:
2022-10-28 02:43:48,430 - twisted - 274 - ERROR - _maybe_retry_device_resync-852- - Traceback (most recent call last):
2022-10-28 02:43:48,435 - twisted - 274 - ERROR - _maybe_retry_device_resync-852- - File "/home/synapse/env-pyston-poetry/lib/python3.8-pyston2.3/site-packages/ijson/backends/python.py", line 46, in utf8_encoder
2022-10-28 02:43:48,440 - twisted - 274 - ERROR - _maybe_retry_device_resync-852- - target.close()
2022-10-28 02:43:48,443 - twisted - 274 - ERROR - _maybe_retry_device_resync-852- - File "/home/synapse/env-pyston-poetry/lib/python3.8-pyston2.3/site-packages/ijson/backends/python.py", line 116, in Lexer
2022-10-28 02:43:48,447 - twisted - 274 - ERROR - _maybe_retry_device_resync-852- - target.send(EOF)
2022-10-28 02:43:48,450 - twisted - 274 - ERROR - _maybe_retry_device_resync-852- - File "/home/synapse/env-pyston-poetry/lib/python3.8-pyston2.3/site-packages/ijson/backends/python.py", line 161, in parse_value
2022-10-28 02:43:48,453 - twisted - 274 - ERROR - _maybe_retry_device_resync-852- - raise common.IncompleteJSONError('Incomplete JSON content')
2022-10-28 02:43:48,456 - twisted - 274 - ERROR - _maybe_retry_device_resync-852- - ijson.common.IncompleteJSONError: Incomplete JSON content
```

Contributor guide

Open the contributing guide

Research direction

Start with the _maybe_retry_device_resync path named in the log and trace where ijson parses the response. Reproduce or inspect the IncompleteJSONError handling at that call site and identify the other affected locations; done means expected parsing failures no longer generate repeated log or Sentry noise while genuine failures remain visible.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.