relay: events with embedded NUL (\u0000) reach Postgres and fail with UTF8 0x00 internal error instead of clean rejection
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
**What happens**
Production relay pods (bb-public cell) intermittently log:
```
Internal error: error: database error: database error: error returned from database: invalid byte sequence for encoding "UTF8": 0x00 at line 1870
```
27 occurrences over the last 14 days (first sighting 2026-07-22, ~5/day since, most recent 2026-07-27 12:14Z), across many different pods/hosts, often in pairs seconds apart — consistent with a client retrying after a 500. Datadog: https://app.datadoghq.com/logs?query=kube_cluster_name%3Aproduction-bke-bb-public-infra-cell-01-us-west-2%20kube_namespace%3Abuzz%20%22invalid%20byte%20sequence%22
**Likely root cause**
JSON permits `\u0000` escapes in strings, but Postgres `text`/`jsonb` columns cannot store NUL bytes. An event whose content (or a tag value) contains an embedded NUL passes JSON parsing and signature checks, then blows up at insert time as a Postgres encoding error, which surfaces as `Internal error` (500 / generic failure) to the client instead of a clean `invalid:` rejection.
**Expected**
Events containing NUL bytes in content or tag values should be rejected at validation time (`invalid: content contains disallowed control characters` or similar), before any DB write — not surfaced as a database internal error.
**Notes**
- Blast radius today is one client request per hit (no panic/restart, the transaction just fails), but each hit is a 500 to a real user and an error-stream pollution source during launch monitoring.
- Worth checking all ingest text paths (event content, tags, profile fields, workflow definitions) for the same gap, since the error doesn't identify which statement fired.
Contributor guide
Assessment
This issue has not been assessed yet.