influxdata / influxdata/influxdb

InfluxDB 2.4.0 runtime error at startup

Open
#23,700 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Actual behavior:__
running for a couple of weeks, InfluxDB suddenly stops and fails to start with "panic: runtime error: slice bounds out of range [:1073741893] with length 1073741824"

__Environment info:__

* System info:
Windows Server 2012 R2 Standard
* InfluxDB version:
InfluxDB v2.4.0 (git: de247bab08) build_date: 2022-08-18T19:41:41Z
* Other relevant environment details: Container runtime, disk info, etc
Disk : NTFS, 300+ Free space
RAM : 48GB

__Config:__
{
"assets-path": "",
"bolt-path": "C:\\influxdb\\data\\influxd.bolt",
"e2e-testing": false,
"engine-path": "C:\\influxdb\\data\\engine",
"feature-flags": null,
"flux-log-enabled": false,
"hardening-enabled": false,
"http-bind-address": ":8086",
"http-idle-timeout": 180000000000,
"http-read-header-timeout": 10000000000,
"http-read-timeout": 0,
"http-write-timeout": 0,
"influxql-max-select-buckets": 0,
"influxql-max-select-point": 0,
"influxql-max-select-series": 0,
"instance-id": "",
"log-level": "info",
"metrics-disabled": false,
"nats-max-payload-bytes": 0,
"nats-port": 0,
"no-tasks": false,
"pprof-disabled": false,
"query-concurrency": 1024,
"query-initial-memory-bytes": 0,
"query-max-memory-bytes": 0,
"query-memory-bytes": 0,
"query-queue-size": 1024,
"reporting-disabled": false,
"secret-store": "bolt",
"session-length": 60,
"session-renew-disabled": false,
"sqlite-path": "C:\\influxdb\\data\\influxd.sqlite",
"storage-cache-max-memory-size": 1073741824,
"storage-cache-snapshot-memory-size": 26214400,
"storage-cache-snapshot-write-cold-duration": "10m0s",
"storage-compact-full-write-cold-duration": "4h0m0s",
"storage-compact-throughput-burst": 50331648,
"storage-max-concurrent-compactions": 0,
"storage-max-index-log-file-size": 1048576,
"storage-no-validate-field-size": false,
"storage-retention-check-interval": "30m0s",
"storage-series-file-max-concurrent-snapshot-compactions": 0,
"storage-series-id-set-cache-size": 0,
"storage-shard-precreator-advance-period": "30m0s",
"storage-shard-precreator-check-interval": "10m0s",
"storage-tsm-use-madv-willneed": false,
"storage-validate-keys": false,
"storage-wal-fsync-delay": "0s",
"storage-wal-max-concurrent-writes": 0,
"storage-wal-max-write-delay": 600000000000,
"storage-write-timeout": 10000000000,
"store": "disk",
"testing-always-allow-setup": false,
"tls-cert": "",
"tls-key": "",
"tls-min-version": "1.2",
"tls-strict-ciphers": false,
"tracing-type": "",
"ui-disabled": false,
"vault-addr": "",
"vault-cacert": "",
"vault-capath": "",
"vault-client-cert": "",
"vault-client-key": "",
"vault-client-timeout": 0,
"vault-max-retries": 0,
"vault-skip-verify": false,
"vault-tls-server-name": "",
"vault-token": ""
}

__Logs:__
ts=2022-09-01T10:46:39.710647Z lvl=info msg="Welcome to InfluxDB" log_id=0cfAPTNW000 version=v2.4.0 commit=de247bab08 build_date=2022-08-18T19:41:41Z log_level=info
ts=2022-09-01T10:46:39.721651Z lvl=info msg="Resources opened" log_id=0cfAPTNW000 service=bolt path=C:\\influxdb\\data\\influxd.bolt
ts=2022-09-01T10:46:39.721651Z lvl=info msg="Resources opened" log_id=0cfAPTNW000 service=sqlite path=C:\\influxdb\\data\\influxd.sqlite
ts=2022-09-01T10:46:39.738648Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0cfAPTNW000 bolt_path=C:\\influxdb\\data\\influxd.bolt
ts=2022-09-01T10:46:39.738648Z lvl=info msg="Using data dir" log_id=0cfAPTNW000 service=storage-engine service=store path=C:\\influxdb\\data\\engine\\data
ts=2022-09-01T10:46:39.739651Z lvl=info msg="Compaction settings" log_id=0cfAPTNW000 service=storage-engine service=store max_concurrent_compactions=4 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
ts=2022-09-01T10:46:39.739651Z lvl=info msg="Open store (start)" log_id=0cfAPTNW000 service=storage-engine service=store op_name=tsdb_open op_event=start
ts=2022-09-01T10:46:41.112747Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi
ts=2022-09-01T10:46:41.113746Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi
ts=2022-09-01T10:46:41.114747Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi
ts=2022-09-01T10:46:41.114747Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi
ts=2022-09-01T10:46:41.115748Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\3e1f65ab0ae9abf9\\autogen\\708\\000000024-000000003.tsm id=0 duration=1.001ms
ts=2022-09-01T10:46:41.116747Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\3e1f65ab0ae9abf9\\autogen\\869\\000000008-000000002.tsm id=0 duration=0.999ms
ts=2022-09-01T10:46:41.116747Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\3e1f65ab0ae9abf9\\autogen\\869\\000000017-000000001.tsm id=2 duration=0.000ms
ts=2022-09-01T10:46:41.116747Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\3e1f65ab0ae9abf9\\autogen\\869\\000000016-000000002.tsm id=1 duration=0.000ms
ts=2022-09-01T10:46:41.116747Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\3e1f65ab0ae9abf9\\autogen\\761\\000000024-000000003.tsm id=0 duration=0.000ms
ts=2022-09-01T10:46:41.116747Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\3e1f65ab0ae9abf9\\autogen\\869\\000000019-000000001.tsm id=4 duration=0.000ms
ts=2022-09-01T10:46:41.116747Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\3e1f65ab0ae9abf9\\autogen\\869\\000000018-000000001.tsm id=3 duration=0.000ms
ts=2022-09-01T10:46:41.117746Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\3e1f65ab0ae9abf9\\autogen\\869\\000000020-000000001.tsm id=5 duration=0.999ms
ts=2022-09-01T10:46:41.117746Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\3e1f65ab0ae9abf9\\autogen\\816\\000000024-000000003.tsm id=0 duration=0.000ms
ts=2022-09-01T10:46:41.120755Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi
ts=2022-09-01T10:46:41.122749Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi
ts=2022-09-01T10:46:41.213752Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\394\\000012517-000000007.tsm id=3 duration=91.002ms
ts=2022-09-01T10:46:41.272757Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\763\\000004987-000000001.tsm id=11 duration=144.003ms
ts=2022-09-01T10:46:41.275762Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\763\\000004736-000000006.tsm id=1 duration=150.008ms
ts=2022-09-01T10:46:41.491776Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\394\\000012517-000000004.tsm id=0 duration=370.023ms
ts=2022-09-01T10:46:41.734035Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi
ts=2022-09-01T10:46:41.737030Z lvl=info msg="Reading file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=cacheloader path=C:\\influxdb\\data\\engine\\wal\\5ae3c808ebcb8ea7\\autogen\\871\\_00001.wal size=4702848
ts=2022-09-01T10:46:41.775452Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi
ts=2022-09-01T10:46:41.844666Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\394\\000012517-000000006.tsm id=2 duration=721.908ms
ts=2022-09-01T10:46:41.844666Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\763\\000004736-000000005.tsm id=0 duration=718.904ms
ts=2022-09-01T10:46:41.924666Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\394\\000012517-000000005.tsm id=1 duration=801.907ms
ts=2022-09-01T10:46:41.960669Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\763\\000004928-000000003.tsm id=4 duration=687.903ms
ts=2022-09-01T10:46:42.021676Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\763\\000004985-000000001.tsm id=9 duration=97.009ms
ts=2022-09-01T10:46:42.044678Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\58ca5e890d8e8fc9\\autogen\\858\\000000012-000000002.tsm id=0 duration=23.002ms
ts=2022-09-01T10:46:42.061676Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\763\\000004896-000000003.tsm id=3 duration=847.913ms
ts=2022-09-01T10:46:42.070675Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\763\\000004986-000000001.tsm id=10 duration=109.001ms
ts=2022-09-01T10:46:42.122685Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\763\\000004960-000000003.tsm id=5 duration=846.912ms
ts=2022-09-01T10:46:42.144685Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\763\\000004864-000000004.tsm id=2 duration=1008.924ms
ts=2022-09-01T10:46:42.144685Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\763\\000004968-000000002.tsm id=6 duration=651.906ms
ts=2022-09-01T10:46:42.192693Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\763\\000004976-000000002.tsm id=7 duration=348.023ms
ts=2022-09-01T10:46:42.200689Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\\influxdb\\data\\engine\\data\\14aba2a311644add\\autogen\\763\\000004984-000000002.tsm id=8 duration=356.018ms
ts=2022-09-01T10:46:42.200689Z lvl=info msg="Reading file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=cacheloader path=C:\\influxdb\\data\\engine\\wal\\14aba2a311644add\\autogen\\763\\_04988.wal size=3187524
ts=2022-09-01T10:46:45.694932Z lvl=info msg="Open store (end)" log_id=0cfAPTNW000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=5955.205ms

panic: runtime error: slice bounds out of range [:1073741893] with length 1073741824

goroutine 1 [running]:
github.com/influxdata/influxdb/v2/pkg/mmap.Map({0xc0015ca200, 0x3e}, 0x0)
/root/project/pkg/mmap/mmap_windows.go:45 +0x42e
github.com/influxdata/influxdb/v2/tsdb.(*SeriesIndex).Open.func1(0xc000ad95e0)
/root/project/tsdb/series_index.go:69 +0x9f
github.com/influxdata/influxdb/v2/tsdb.(*SeriesIndex).Open(0xc000ad95e0)
/root/project/tsdb/series_index.go:84 +0x25
github.com/influxdata/influxdb/v2/tsdb.(*SeriesPartition).Open.func1(0xc0055f84d0)
/root/project/tsdb/series_partition.go:88 +0x10a
github.com/influxdata/influxdb/v2/tsdb.(*SeriesPartition).Open(0xc0055f84d0)
/root/project/tsdb/series_partition.go:95 +0x6f
github.com/influxdata/influxdb/v2/tsdb.(*SeriesFile).Open(0xc004b7ca00)
/root/project/tsdb/series_file.go:91 +0x4d5
github.com/influxdata/influxdb/v2/tsdb.(*Store).openSeriesFile(0xc000a3b800, {0xc001b79230, 0x10})
/root/project/tsdb/store.go:556 +0x11a
github.com/influxdata/influxdb/v2/tsdb.(*Store).loadShards(0xc000a3b800, {0x72ca1a8?, 0xc001a5d900})
/root/project/tsdb/store.go:363 +0x12c7
github.com/influxdata/influxdb/v2/tsdb.(*Store).Open(0xc000a3b800, {0x72ca1a8, 0xc001a5d900})
/root/project/tsdb/store.go:261 +0x226
github.com/influxdata/influxdb/v2/storage.(*Engine).Open(0xc001bc2180, {0x72ca1a8, 0xc001a5d900})
/root/project/storage/engine.go:195 +0x11f
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.(*Launcher).run(0xc000e38380, {0x72ca1a8?, 0xc001a5d8c0?}, 0xc000868000)
/root/project/cmd/influxd/launcher/launcher.go:345 +0x2335
github.com/influxdata/influxdb/v2/cmd/influxd/launcher.cmdRunE.func1()
/root/project/cmd/influxd/launcher/cmd.go:124 +0x2da
github.com/influxdata/influxdb/v2/kit/cli.NewCommand.func1(0xc000169b80?, {0x8bbc008?, 0x0?, 0x0?})
/root/project/kit/cli/viper.go:54 +0x1e
github.com/spf13/cobra.(*Command).execute(0xc000169b80, {0xc000108200, 0x0, 0x0})
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x67c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000169b80)
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x39c
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
main.main()
/root/project/cmd/influxd/main.go:61 +0x4b8

Contributor guide

Open the contributing guide

Research direction

Start with the storage-engine startup logs and the WAL/TSM files under the reported data directory. Reproduce the startup failure with the affected data and trace the slice-bounds panic during recovery. Done means the existing data starts successfully without the runtime panic, with a regression test covering the failure case.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.