element-hq / element-hq/dendrite

[DRAFT] Postgres ERROR: invalid byte sequence for encoding "UTF8": 0x00

Open
#2,595 1 comment 0 reactions 0 assignees View on GitHub
C-Keyserver C-Postgres C-Roomserver changes database schema O-Uncommon S-Major T-Defect
Dominant language
Go
Stars
965
Forks
101
PR merge metrics
No merged PRs in 30d

Description

*This issue was originally created by [**@mibmo**](https://github.com/mibmo) at .*

## Background information

- **Dendrite version or git SHA**: 0.8.9
- **Monolith or Polylith?**: Polylith
- **SQLite3 or Postgres?**: Postgres
- **Running in Docker?**: no, Kubernetes v1.23.6.
- **`go version`**: ` n/a
- **Client used (if applicable)**: n/a

I used the following image: `docker.io/matrixdotorg/dendrite-polylith@sha256:43816ad9050e9cb18b8e5c0cd41f5a779cec295fb122f164b4461f998ec5d344`
Dendrite was split over 6 nodes.

### Description

#### What
Databases started spewing errors about

#### Who
Entire homeserver.

#### How
Databases emitting `invalid byte sequence for encoding "UTF8": 0x00` (and similar) errors, and components writing `ERROR: relation "?" does not exist at character ?`

#### When
I'm not 100% sure. Although I think it had already begun happening prior, it started becoming a major issue after I tried to set my avatar to an animated one (webp) and the server crashed (a user of the homeserver wrote to me "the server bricked after you changed your profile picture", and I experienced the same). After the components that crashed came back up again, I checked the logs and gradually the Postgres instances began throwing errors as described above. I'm not sure which exact components crashed, but considering we were seeing the "network lost" messages in our clients, probably the client api? (and more?)

### Steps to reproduce

I am unsure how to reproduce this at the moment. I am filing this issue as I was requested to, so it does not currently have reproduction steps. I understand if the issue is closed; this is just for reference :)

I am however suspicious of database locale. With Postgres locale set to en_US, the errors occured, but upon changing to `en_GB` (same as `dendrite.matrix.org`) the errors stopped. It is however _completely_ possible that this is unrelated, but I don't have enough familiarity with the internals of Postgres nor Dendrite to say for certain. :)

I first noticed everything with the keyserver database:
![image](https://user-images.githubusercontent.com/87388017/181088901-1275646f-7f60-46d4-a62b-fd80a532dc04.png)
![image](https://user-images.githubusercontent.com/87388017/181089068-3fea262e-4be7-45ba-bd3a-f94c80ceaf46.png)

Link to where I ask in #dendrite:matrix.org for help: https://matrix.to/#/!RcWPWcZrMeBxOGaalX:matrix.org/$mWcIimXJ33UNl2spsYu_-tBqMKjoPXQ4ro3_op3rP7s?via=matrix.org&via=envs.net&via=kanp.ai

(apologies for it being images. I didn't copy the relevant lines to a text document before troubleshooting the issue, and no longer have access to the raw logs)

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or reproducible steps are identified. Start by reviewing the Postgres errors and the components that crashed after the avatar change, then compare the en_US and en_GB database configurations. Done means identifying a reproducible cause for the invalid UTF8 and missing-relation errors and defining the affected Dendrite component or database operation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.