influxdata / influxdata/influxdb

tsm.bad files after restart and .tsm missing

Open
#23,267 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

1.x
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Just noticed that about a month worth of data is missing from one of databases.

Screenshot 2022-04-11 at 09 18 12

Running on `Raspberry Pi 4 4GB`
`Raspbian GNU/Linux 10 (buster)`
`InfluxDB shell version: 1.6.4`

Shards are there:
```name: smartshunt
id database retention_policy shard_group start_time end_time expiry_time owners
-- -------- ---------------- ----------- ---------- -------- ----------- ------
356 smartshunt autogen 356 2021-11-08T00:00:00Z 2021-11-15T00:00:00Z 2021-11-15T00:00:00Z
360 smartshunt autogen 360 2021-11-15T00:00:00Z 2021-11-22T00:00:00Z 2021-11-22T00:00:00Z
370 smartshunt autogen 370 2021-11-22T00:00:00Z 2021-11-29T00:00:00Z 2021-11-29T00:00:00Z
380 smartshunt autogen 380 2021-11-29T00:00:00Z 2021-12-06T00:00:00Z 2021-12-06T00:00:00Z
390 smartshunt autogen 390 2021-12-06T00:00:00Z 2021-12-13T00:00:00Z 2021-12-13T00:00:00Z
400 smartshunt autogen 400 2021-12-13T00:00:00Z 2021-12-20T00:00:00Z 2021-12-20T00:00:00Z
410 smartshunt autogen 410 2021-12-20T00:00:00Z 2021-12-27T00:00:00Z 2021-12-27T00:00:00Z
420 smartshunt autogen 420 2021-12-27T00:00:00Z 2022-01-03T00:00:00Z 2022-01-03T00:00:00Z
430 smartshunt autogen 430 2022-01-03T00:00:00Z 2022-01-10T00:00:00Z 2022-01-10T00:00:00Z
440 smartshunt autogen 440 2022-01-10T00:00:00Z 2022-01-17T00:00:00Z 2022-01-17T00:00:00Z
450 smartshunt autogen 450 2022-01-17T00:00:00Z 2022-01-24T00:00:00Z 2022-01-24T00:00:00Z
460 smartshunt autogen 460 2022-01-24T00:00:00Z 2022-01-31T00:00:00Z 2022-01-31T00:00:00Z
470 smartshunt autogen 470 2022-01-31T00:00:00Z 2022-02-07T00:00:00Z 2022-02-07T00:00:00Z
480 smartshunt autogen 480 2022-02-07T00:00:00Z 2022-02-14T00:00:00Z 2022-02-14T00:00:00Z
490 smartshunt autogen 490 2022-02-14T00:00:00Z 2022-02-21T00:00:00Z 2022-02-21T00:00:00Z
500 smartshunt autogen 500 2022-02-21T00:00:00Z 2022-02-28T00:00:00Z 2022-02-28T00:00:00Z
510 smartshunt autogen 510 2022-02-28T00:00:00Z 2022-03-07T00:00:00Z 2022-03-07T00:00:00Z
520 smartshunt autogen 520 2022-03-07T00:00:00Z 2022-03-14T00:00:00Z 2022-03-14T00:00:00Z
530 smartshunt autogen 530 2022-03-14T00:00:00Z 2022-03-21T00:00:00Z 2022-03-21T00:00:00Z
540 smartshunt autogen 540 2022-03-21T00:00:00Z 2022-03-28T00:00:00Z 2022-03-28T00:00:00Z
550 smartshunt autogen 550 2022-03-28T00:00:00Z 2022-04-04T00:00:00Z 2022-04-04T00:00:00Z
560 smartshunt autogen 560 2022-04-04T00:00:00Z 2022-04-11T00:00:00Z 2022-04-11T00:00:00Z
570 smartshunt autogen 570 2022-04-11T00:00:00Z 2022-04-18T00:00:00Z 2022-04-18T00:00:00Z
```

But went to files directly it was like that:
```
-rw-r--r-- 1 influxdb influxdb 15667654 Mar 20 23:10 000000003-000000002.tsm.bad
-rw-r--r-- 1 influxdb influxdb 109 Apr 11 10:19 fields.idx
```

Also went from `inmem` to `tsi1` but no luck.
Any way to fix it?

_Originally posted by @jeronimo in https://github.com/influxdata/influxdb/issues/14180#issuecomment-1095209544_

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report identifies a Raspberry Pi installation, InfluxDB 1.6.4, a .tsm.bad file, and a missing .tsm file. Start by examining the affected shard files and the TSM and tsi1 storage paths; done means determining why the file became invalid and documenting or implementing a verified recovery path without further data loss.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, raspberry-pi
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.