influxdata / influxdata/influxdb
Failure to allocate memory marks data as corrupt
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Related to #23246
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
0. Verify the virtual memory limit per process is set to `65530` by running `cat /sys/proc/vm/max_map_count`
1. Get +68TB of data in your database
2. Restart influx service
3. Wait until the database is almost loaded
4. Crash while opening index files with the message `failed to allocate memory`
__Expected behavior:__
The influx process starts normally and _does not mark valid data files as corrupt_
__Actual behavior:__
The Influx process fails to start because the maximum number of virtual memory pages is reached. Data is marked as corrupt, even though it is not. Influx would eventually start because it does not try to load `*.tsm.bad` files, but this would also make the data inaccessible.
__Environment info:__
* System info: Run `uname -srm` and copy the output here
* Linux 5.15.0-58-generic x86_64
* InfluxDB version: Run `influxd version` and copy the output here
* 2.6.1
* Other relevant environment details: Container runtime, disk info, etc:
* A database of +80TB accross 5 buckets
__Config:__
Copy any non-default config values here or attach the full config as a gist or file.
__Fix:__
Place a check here [in `tsdb/engine/tsm1/file_store.go:664`](https://github.com/influxdata/influxdb/blob/8ed55e72b85b0807c97d3d1d7d12e9f07c9737e9/tsdb/engine/tsm1/file_store.go#L611) that checks whether or not the problem is that memory cannot be allocated. If that it the issue, do not mark the file as bad.
__Logs:__
Include snippet of errors in log.
Logs
```
ts=2023-01-25T18:20:49.912435Z lvl=info msg="Welcome to InfluxDB" log_id=0fbYQZU0000 version=v2.6.1 commit=9dcf880fe0 build_date=2023-01-25T18:20:49Z log_level=debug
ts=2023-01-25T18:20:49.912521Z lvl=debug msg="loaded config file" log_id=0fbYQZU0000 path=/etc/influxdb/config.toml
ts=2023-01-25T18:20:49.913335Z lvl=info msg="Resources opened" log_id=0fbYQZU0000 service=bolt path=/srv/influxdb/server-2/influxd.bolt
ts=2023-01-25T18:20:49.913449Z lvl=info msg="Resources opened" log_id=0fbYQZU0000 service=sqlite path=/srv/influxdb/server-2/influxd.sqlite
ts=2023-01-25T18:20:49.918680Z lvl=debug msg="buckets find" log_id=0fbYQZU0000 store=new took=0.354ms
ts=2023-01-25T18:20:49.918722Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0fbYQZU0000 bolt_path=/srv/influxdb/server-2/influxd.bolt
ts=2023-01-25T18:20:49.918851Z lvl=info msg="Using data dir" log_id=0fbYQZU0000 service=storage-engine service=store path=/srv/influxdb/server-2/engine/data
ts=2023-01-25T18:20:49.918880Z lvl=info msg="Compaction settings" log_id=0fbYQZU0000 service=storage-engine service=store max_concurrent_compactions=16 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
ts=2023-01-25T18:20:49.918894Z lvl=info msg="Open store (start)" log_id=0fbYQZU0000 service=storage-engine service=store op_name=tsdb_open op_event=start
ts=2023-01-25T18:20:49.968315Z lvl=info msg="index opened with 8 partitions" log_id=0fbYQZU0000 service=storage-engine index=tsi
ts=2023-01-25T18:20:49.969559Z lvl=info msg="index opened with 8 partitions" log_id=0fbYQZU0000 service=storage-engine index=tsi
ts=2023-01-25T18:20:49.969581Z lvl=info msg="index opened with 8 partitions" log_id=0fbYQZU0000 service=storage-engine index=tsi
...
ts=2023-01-25T18:21:08.298327Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/64655d4f6f6e4c60/autogen/1212/000004480-000000006.tsm id=17 duration=1.625ms
ts=2023-01-25T18:21:08.299641Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/64655d4f6f6e4c60/autogen/1212/000004480-000000007.tsm id=18 duration=2.773ms
ts=2023-01-25T18:21:08.299787Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/64655d4f6f6e4c60/autogen/1212/000005376-000000005.tsm id=20 duration=1.424ms
ts=2023-01-25T18:21:08.300023Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/64655d4f6f6e4c60/autogen/1212/000004480-000000008.tsm id=19 duration=2.590ms
ts=2023-01-25T18:21:08.301218Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/64655d4f6f6e4c60/autogen/1212/000005376-000000007.tsm id=22 duration=1.407ms
ts=2023-01-25T18:21:08.301803Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/64655d4f6f6e4c60/autogen/1154/000005376-000000005.tsm id=20 duration=41.650ms
...
ts=2023-01-25T18:41:58.373553Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000055552-000000005.tsm id=183 duration=1102.175ms
ts=2023-01-25T18:41:58.425901Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000058240-000000007.tsm id=194 duration=759.708ms
ts=2023-01-25T18:41:58.450058Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000054656-000000006.tsm id=181 duration=1190.872ms
ts=2023-01-25T18:41:58.477517Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000060032-000000007.tsm id=200 duration=596.377ms
ts=2023-01-25T18:41:58.512584Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000053760-000000005.tsm id=177 duration=1308.265ms
ts=2023-01-25T18:41:58.618123Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000033152-000000006.tsm id=109 duration=2412.123ms
ts=2023-01-25T18:41:58.618185Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000033152-000000006.tsm id=109 error="cannot allocate memory"
ts=2023-01-25T18:41:58.628056Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000059136-000000005.tsm id=195 duration=870.848ms
ts=2023-01-25T18:41:58.628118Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000059136-000000005.tsm id=195 error="cannot allocate memory"
ts=2023-01-25T18:41:58.632281Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000056448-000000007.tsm id=188 duration=1211.951ms
ts=2023-01-25T18:41:58.632342Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000056448-000000007.tsm id=188 error="cannot allocate memory"
ts=2023-01-25T18:41:58.632796Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000060032-000000006.tsm id=199 duration=779.674ms
ts=2023-01-25T18:41:58.632860Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000060032-000000006.tsm id=199 error="cannot allocate memory"
ts=2023-01-25T18:41:58.633198Z lvl=error msg="Failed to open shard" log_id=0fbYQZU0000 service=storage-engine service=store op_name=tsdb_open db_shard_id=2316 error="[shard 2316] cannot read corrupt file /srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000033152-000000006.tsm: cannot allocate memory"
ts=2023-01-25T18:41:58.637849Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000059136-000000007.tsm id=197 duration=827.411ms
ts=2023-01-25T18:41:58.637915Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000059136-000000007.tsm id=197 error="cannot allocate memory"
ts=2023-01-25T18:41:58.651931Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000060928-000000006.tsm id=202 duration=711.669ms
ts=2023-01-25T18:41:58.651993Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000060928-000000006.tsm id=202 error="cannot allocate memory"
ts=2023-01-25T18:41:58.674625Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000065408-000000005.tsm id=216 duration=197.010ms
ts=2023-01-25T18:41:58.674690Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000065408-000000005.tsm id=216 error="cannot allocate memory"
ts=2023-01-25T18:41:58.680297Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000065408-000000007.tsm id=218 duration=47.779ms
ts=2023-01-25T18:41:58.680364Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000065408-000000007.tsm id=218 error="cannot allocate memory"
ts=2023-01-25T18:41:58.701011Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000061824-000000007.tsm id=206 duration=661.573ms
ts=2023-01-25T18:41:58.701077Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000061824-000000007.tsm id=206 error="cannot allocate memory"
ts=2023-01-25T18:41:58.728106Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000061824-000000006.tsm id=205 duration=713.875ms
ts=2023-01-25T18:41:58.728170Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000061824-000000006.tsm id=205 error="cannot allocate memory"
ts=2023-01-25T18:41:58.734582Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000060928-000000007.tsm id=203 duration=745.129ms
ts=2023-01-25T18:41:58.734644Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000060928-000000007.tsm id=203 error="cannot allocate memory"
ts=2023-01-25T18:41:58.762078Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000062720-000000007.tsm id=209 duration=653.349ms
ts=2023-01-25T18:41:58.762140Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000062720-000000007.tsm id=209 error="cannot allocate memory"
ts=2023-01-25T18:41:58.803846Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000057344-000000006.tsm id=190 duration=1325.792ms
ts=2023-01-25T18:41:58.803909Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000057344-000000006.tsm id=190 error="cannot allocate memory"
ts=2023-01-25T18:41:58.848275Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000062720-000000005.tsm id=207 duration=753.574ms
ts=2023-01-25T18:41:58.848342Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000062720-000000005.tsm id=207 error="cannot allocate memory"
ts=2023-01-25T18:41:58.850634Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000060928-000000005.tsm id=201 duration=924.362ms
ts=2023-01-25T18:41:58.850698Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000060928-000000005.tsm id=201 error="cannot allocate memory"
ts=2023-01-25T18:41:58.852278Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000061824-000000005.tsm id=204 duration=851.501ms
ts=2023-01-25T18:41:58.852341Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000061824-000000005.tsm id=204 error="cannot allocate memory"
ts=2023-01-25T18:41:58.876393Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000063616-000000007.tsm id=212 duration=610.106ms
ts=2023-01-25T18:41:58.876466Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000063616-000000007.tsm id=212 error="cannot allocate memory"
ts=2023-01-25T18:41:58.926520Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000064512-000000007.tsm id=215 duration=476.373ms
ts=2023-01-25T18:41:58.926582Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000064512-000000007.tsm id=215 error="cannot allocate memory"
ts=2023-01-25T18:41:58.946339Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000062720-000000006.tsm id=208 duration=846.386ms
ts=2023-01-25T18:41:58.946402Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000062720-000000006.tsm id=208 error="cannot allocate memory"
ts=2023-01-25T18:41:58.947688Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000054656-000000005.tsm id=180 duration=1691.661ms
ts=2023-01-25T18:41:58.947752Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000054656-000000005.tsm id=180 error="cannot allocate memory"
ts=2023-01-25T18:41:58.975869Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000060032-000000005.tsm id=198 duration=1150.312ms
ts=2023-01-25T18:41:58.975931Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000060032-000000005.tsm id=198 error="cannot allocate memory"
ts=2023-01-25T18:41:58.984079Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000058240-000000005.tsm id=192 duration=1464.107ms
ts=2023-01-25T18:41:58.984151Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000058240-000000005.tsm id=192 error="cannot allocate memory"
ts=2023-01-25T18:41:59.004448Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000059136-000000006.tsm id=196 duration=1242.293ms
ts=2023-01-25T18:41:59.004520Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000059136-000000006.tsm id=196 error="cannot allocate memory"
ts=2023-01-25T18:41:59.012637Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000064512-000000006.tsm id=214 duration=586.639ms
ts=2023-01-25T18:41:59.012692Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000064512-000000006.tsm id=214 error="cannot allocate memory"
ts=2023-01-25T18:41:59.013067Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000038528-000000006.tsm id=127 duration=1868.599ms
ts=2023-01-25T18:41:59.013132Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000038528-000000006.tsm id=127 error="cannot allocate memory"
ts=2023-01-25T18:41:59.030590Z lvl=error msg="Failed to open shard" log_id=0fbYQZU0000 service=storage-engine service=store op_name=tsdb_open db_shard_id=2839 error="[shard 2839] cannot allocate memory"
ts=2023-01-25T18:41:59.035460Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000063616-000000005.tsm id=210 duration=857.893ms
ts=2023-01-25T18:41:59.035512Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000063616-000000005.tsm id=210 error="cannot allocate memory"
ts=2023-01-25T18:41:59.060048Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000077056-000000006.tsm id=256 duration=1911.747ms
ts=2023-01-25T18:41:59.060112Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000077056-000000006.tsm id=256 error="cannot allocate memory"
ts=2023-01-25T18:41:59.062404Z lvl=error msg="Failed to open shard" log_id=0fbYQZU0000 service=storage-engine service=store op_name=tsdb_open db_shard_id=3103 error="[shard 3103] cannot allocate memory"
ts=2023-01-25T18:41:59.081984Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000058240-000000006.tsm id=193 duration=1427.590ms
ts=2023-01-25T18:41:59.082046Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000058240-000000006.tsm id=193 error="cannot allocate memory"
ts=2023-01-25T18:41:59.095927Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000064512-000000005.tsm id=213 duration=722.275ms
ts=2023-01-25T18:41:59.095984Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000064512-000000005.tsm id=213 error="cannot allocate memory"
ts=2023-01-25T18:41:59.100023Z lvl=error msg="Failed to open shard" log_id=0fbYQZU0000 service=storage-engine service=store op_name=tsdb_open db_shard_id=3055 error="[shard 3055] cannot allocate memory"
ts=2023-01-25T18:41:59.116188Z lvl=error msg="Failed to open shard" log_id=0fbYQZU0000 service=storage-engine service=store op_name=tsdb_open db_shard_id=3079 error="[shard 3079] cannot allocate memory"
ts=2023-01-25T18:41:59.140488Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000063616-000000006.tsm id=211 duration=957.209ms
ts=2023-01-25T18:41:59.140550Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000063616-000000006.tsm id=211 error="cannot allocate memory"
ts=2023-01-25T18:41:59.149880Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000055552-000000006.tsm id=184 duration=1785.417ms
ts=2023-01-25T18:41:59.149943Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000055552-000000006.tsm id=184 error="cannot allocate memory"
ts=2023-01-25T18:41:59.154873Z lvl=error msg="Failed to open shard" log_id=0fbYQZU0000 service=storage-engine service=store op_name=tsdb_open db_shard_id=3175 error="[shard 3175] cannot allocate memory"
ts=2023-01-25T18:41:59.179055Z lvl=error msg="Failed to open shard" log_id=0fbYQZU0000 service=storage-engine service=store op_name=tsdb_open db_shard_id=3127 error="[shard 3127] cannot allocate memory"
ts=2023-01-25T18:41:59.179082Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000065408-000000006.tsm id=217 duration=666.401ms
ts=2023-01-25T18:41:59.179146Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000065408-000000006.tsm id=217 error="cannot allocate memory"
ts=2023-01-25T18:41:59.208678Z lvl=error msg="Failed to open shard" log_id=0fbYQZU0000 service=storage-engine service=store op_name=tsdb_open db_shard_id=3199 error="[shard 3199] cannot allocate memory"
ts=2023-01-25T18:41:59.216185Z lvl=info msg="Opened file" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000032256-000000007.tsm id=107 duration=3027.267ms
ts=2023-01-25T18:41:59.216247Z lvl=error msg="Cannot read corrupt tsm file, renaming" log_id=0fbYQZU0000 service=storage-engine engine=tsm1 service=filestore path=/srv/influxdb/server-2/engine/data/95d019adc278655d/autogen/2316/000032256-000000007.tsm id=107 error="cannot allocate memory"
```
Contributor guide
Research direction
Read tsdb/engine/tsm1/file_store.go around line 664 and trace the TSM opening path that logs "cannot allocate memory". Done means valid TSM files are not renamed as corrupt when memory allocation fails, and startup does not make that data inaccessible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100