influxdata / influxdata/influxdb
Missing tsm file in shard makes writes fail with (unclear) error message "[shard x] unexpected end of JSON input"
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
1. Harddrive is full preventing influxdb successfully writing data
(log: error="write /var/lib/influxdb/engine/data/e7adcc11582b82f9/autogen/222/index/3/L1-00000001.tsi: no space left on device" path=/var/lib/influxdb/engine/data/e7adcc11582b82f9/autogen/222/index/3/L0-00000001.tsl)
2. You end up with corrupt data/shard
3. Restart influxdb after clean up of harddrive, there is now space to write to
4. There is now a shard without a .tsm/.tsl file
5. Try to write data to the time period of the above mentioned shard
__Expected behavior:__
The data is successfully written (the database will handle the missing .tsm file and do what is necessary to be able to write new data).
__Actual behavior:__
Receive error message
(python): HTTP response body: {"code":"internal error","message":"unexpected error writing points to database: [shard 222] unexpected end of JSON input"}
(log): Aug 16 13:08:05 COMPUTER influxd-systemd-start.sh[224631]: ts=2022-08-16T11:08:05.063037Z lvl=warn msg="Write failed creating shard" log_id=0cLaEUw0000 service=storage-engine service=write shard=222 error="opening shard previously failed with: [shard 222] unexpected end of JSON input"
__Environment info:__
* System info: Linux 5.10.0-10-amd64 x86_64
* InfluxDB version: InfluxDB v2.3.0+SNAPSHOT.090f681737 (git: 090f681737) build_date: 2022-06-16T19:33:50Z
* Other relevant environment details: N/A
__Config:__
Default
__Other info:__
Solved by deleting the shard folder in /engine/data/xxxx/autogen (in my case 222).
Contributor guide
Research direction
Use the reproduction steps and the shard path under /engine/data/xxxx/autogen, focusing on the shard-opening and write path when the .tsm/.tsl file is missing. Confirm the failure shown in the logs, then verify that writing to the affected time period succeeds without requiring deletion of the shard folder.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100