JSON encoder corrupts data, adds random memory contents to end of string
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
## Bug Report
**Describe the bug**
When JSON with a long string gets re-encoded by fluent-bit, several bytes of random garbage gets added to the end of the string. The garbage is 1-15 or 1-4 bytes (depending on the SIMD supported in the host CPU) of fluent-bit's memory -- usually snippets of other log streams flowing through the process.
**To Reproduce**
`fluent-bit.conf`:
```
[SERVICE]
Flush 1
Daemon Off
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Tag mwe
Path input.log
Parser mwe-json
Read_from_Head On
Skip_Empty_Lines On
Buffer_Chunk_Size 256KB
Buffer_Max_Size 20MB
Mem_Buf_Limit 512MB
[OUTPUT]
Name loki
Match mwe
Host 127.0.0.1
Port 3100
Line_Format json
Labels job=mwe
Remove_keys seq
Retry_Limit no_limits
```
`parsers.conf`:
```
[PARSER]
Name mwe-json
Format json
Time_Key at
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
```
Logs, ingesting which reproduces the corruption: [long_unicode_payloads.json](https://github.com/user-attachments/files/30989811/long_unicode_payloads.json)
Observe that each logline has the shape `{"seq":123,"msg":"123,123,...,123,123-------"}` with `.msg` being ~40KB ending in ``:
```sh
jq < long_unicode_payloads.json '.msg[-7:]'
```
A simple script that pretends to be Loki and dumps all pushed payloads to stdout: [loki_http_to_stdout.py](https://github.com/user-attachments/files/30989844/loki_http_to_stdout.py)
The multibyte characters in logs are required for reproduction, and so is a Loki [OUTPUT].
Steps to reproduce the problem:
```sh
PORT=3100 python3 loki_http_to_stdout.py > loki.jsonl &
touch input.log
docker run --rm --network host -v "$PWD:/mwe" -w /mwe fluent/fluent-bit:5.1.0 /fluent-bit/bin/fluent-bit -c fluent-bit.conf &
cat long_unicode_payloads.json > input.log
```
Wait a few seconds for it to settle.
```sh
kill -INT %1 # python3
kill -INT %2 # docker
```
Now inspect the contents of `loki.jsonl`:
```sh
jq < loki.jsonl '.streams[].values[][1] | fromjson | .msg[-32:]'
```
The JSON received by Loki is different: `.msg` now doesn't end in `` but rather has a few bytes added at the end, with infixes of other loglines.
On my machine it looks like this:
```
"0,0,0,0,0,0,0,0,0,0,0,0-6"
"1,1,1,1,1,1,1,1,1,1,1,116"
"2,2,2,2---------------,16"
"3,3,3,3--------------2,2,"
"4,4,4,4-------------,3,3,"
"5,5,5,5------------2,2,2,"
...
"1,191,191----0,190,190,19"
"2,192,192---90,190,190,19"
"3,193,193--192,192,192,19"
"4,194,194-,193,193,193,19"
"5,195,195---------------9"
"6,196,196--------------18"
"7,197,197-------------,18"
"8,198,198------------>,18"
"9,199,199-----------98,19"
```
Some additional pointers for reproducing:
- Only happens for long lines, the shortest corruption I found in our production logs was 24KB.
- The long value has to be the last key inside the JSON object. Lua filters tend to randomize the ordering of keys.
- There have to be multibyte characters inside the string.
- I believe the root case to be [`flb_utils_write_str_escaped`](https://github.com/fluent/fluent-bit/blob/ae51533a2d7a1cb8cb817ded61e28e2c6e88be9a/src/flb_utils.c#L877) incorrectly deciding whether a SIMD batch of bytes can be grabbed before the string ends, and grabbing some bytes past the end of the buffer. Whenever those uninitialized memory bytes are all "no escaping needed" bytes (`>=0x20`, `<0x80`, not `"`, not `\`), the bug reproduces.
- With a simple file output, the JSON seems to be surrounded by msgpack framing in memory (high bit bytes).
- Using a Loki output with `Remove_keys` seems to ensure that the JSON lives in a freshly allocated buffer, potentially reused from a previous log line.
**Expected behavior**
JSON strings should decode to the same as the orignal.
**Your Environment**
* Version used: in production we use 5.0.3, but the above MWE works on 5.1.0 also
* Operating System: x86_64 linux
Contributor guide
Research direction
Start with flb_utils_write_str_escaped in src/flb_utils.c around the linked line, then reproduce with the supplied fluent-bit.conf, parsers.conf, long_unicode_payloads.json, and Loki helper. Check the SIMD end-of-string handling for long multibyte strings. Done means re-encoded JSON strings decode to the original value without appended memory contents.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 62/100