influxdata / influxdata/influxdb
Out of memory on startup - Docker container suddenly keeps restarting
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behaviour.
1. I'm running influxdb 1.8.5 in a docker container for several month now. Raspberry Pi 4 8GB (32bit OS)
2. Suddenly the container keeps restarting every couple of seconds
3. Console in the container says "out of memory" when starting up.
4. I can not intervene with some troubleshooting because the time until the next container restart is very limited.
__Expected behaviour:__
Container keeps on running fine, as it did the last month ;)
__Actual behaviour:__
Container keeps restarting.
__Environment info:__
* influxdb 1.8.5 (docker pull influxdb:1.8.5 from hub.docker.com)
* in a docker container
* Raspberry Pi 4 8GB
* 32bit Raspbian OS
__Config:__
default config
__Logs:__
```
[..]
ts=2023-10-23T21:37:35.190478Z lvl=info msg="Reading file" log_id=0l3cLTiG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1150/_00469.wal size=380278
ts=2023-10-23T21:37:35.287675Z lvl=info msg="Reading file" log_id=0l3cLTiG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1150/_00470.wal size=381883
ts=2023-10-23T21:37:35.384642Z lvl=info msg="Reading file" log_id=0l3cLTiG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1150/_00471.wal size=381139
ts=2023-10-23T21:37:35.502067Z lvl=info msg="Reading file" log_id=0l3cLTiG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1150/_00472.wal size=381268
ts=2023-10-23T21:37:35.648252Z lvl=info msg="Reading file" log_id=0l3cLTiG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1150/_00473.wal size=380549
ts=2023-10-23T21:37:35.796081Z lvl=info msg="Reading file" log_id=0l3cLTiG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1150/_00474.wal size=379749
ts=2023-10-23T21:37:35.944870Z lvl=info msg="Reading file" log_id=0l3cLTiG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1150/_00475.wal size=381238
runtime: out of memory: cannot allocate 8192-byte block (552501248 in use)
fatal error: out of memory
goroutine 66 [running]:
runtime.throw(0xfd9c03, 0xd)
/usr/local/go/src/runtime/panic.go:774 +0x5c fp=0x464f114 sp=0x464f100 pc=0x41644
runtime.(*mcache).refill(0xb6ec26d0, 0x21)
/usr/local/go/src/runtime/mcache.go:140 +0xfc fp=0x464f128 sp=0x464f114 pc=0x262ec
runtime.(*mcache).nextFree(0xb6ec26d0, 0x17025121, 0x1, 0x1d0, 0xcfe890)
/usr/local/go/src/runtime/malloc.go:854 +0x7c fp=0x464f148 sp=0x464f128 pc=0x1b0f4
runtime.mallocgc(0xf0, 0x0, 0x0, 0xc9b6b8)
/usr/local/go/src/runtime/malloc.go:1022 +0x7a0 fp=0x464f1b0 sp=0x464f148 pc=0x1ba40
runtime.rawbyteslice(0xe6, 0x0, 0x0, 0x0)
/usr/local/go/src/runtime/string.go:272 +0x84 fp=0x464f1cc sp=0x464f1b0 pc=0x5f908
runtime.stringtoslicebyte(0x0, 0xbe1585a0, 0xe6, 0xf0, 0xbe0c80c0, 0x1)
/usr/local/go/src/runtime/string.go:161 +0xa4 fp=0x464f1ec sp=0x464f1cc pc=0x5f354
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Cache).WriteMulti(0x46d4200, 0x5754100, 0x464f510, 0x1ea2988)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:343 +0x268 fp=0x464f2d8 sp=0x464f1ec pc=0xc9b674
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*CacheLoader).Load.func1(0x464f594, 0x5314bd0, 0x464f58c, 0x46d4200, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:747 +0x4d4 fp=0x464f568 sp=0x464f2d8 pc=0xd10010
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*CacheLoader).Load(0x5314bd0, 0x46d4200, 0x1, 0x1)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:758 +0x88 fp=0x464f59c sp=0x464f568 pc=0xc9d028
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).reloadCache(0x53b0480, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2392 +0x1d4 fp=0x464f6ec sp=0x464f59c pc=0xccca50
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).Open(0x53b0480, 0x431a700, 0x1ec5508)
/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:754 +0x28c fp=0x464f734 sp=0x464f6ec pc=0xcc37ec
github.com/influxdata/influxdb/tsdb.(*Shard).Open.func1(0x57e6000, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/shard.go:344 +0x298 fp=0x464fa58 sp=0x464f734 pc=0x5ce5c8
github.com/influxdata/influxdb/tsdb.(*Shard).Open(0x57e6000, 0x431a7e0, 0x511c4b0)
/go/src/github.com/influxdata/influxdb/tsdb/shard.go:355 +0x1c fp=0x464fa84 sp=0x464fa58 pc=0x5b70fc
github.com/influxdata/influxdb/tsdb.(*Store).loadShards.func1(0x413b8c0, 0x41aa140, 0x4383050, 0x413b900, 0xfc3018, 0x412e780, 0x4339280, 0x436ea57, 0x9, 0x414eb01, ...)
/go/src/github.com/influxdata/influxdb/tsdb/store.go:404 +0x4c4 fp=0x464ffb4 sp=0x464fa84 pc=0x5cf6e8
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:868 +0x4 fp=0x464ffb4 sp=0x464ffb4 pc=0x73610
created by github.com/influxdata/influxdb/tsdb.(*Store).loadShards
/go/src/github.com/influxdata/influxdb/tsdb/store.go:362 +0xb64
goroutine 1 [chan receive]:
github.com/influxdata/influxdb/tsdb.(*Store).loadShards(0x41aa140, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/store.go:421 +0x115c
github.com/influxdata/influxdb/tsdb.(*Store).Open(0x41aa140, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/store.go:221 +0x1a4
github.com/influxdata/influxdb/cmd/influxd/run.(*Server).Open(0x41a6320, 0x429dcac, 0x41a6320)
/go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:444 +0x894
github.com/influxdata/influxdb/cmd/influxd/run.(*Command).Run(0x43323c0, 0x400a148, 0x2, 0x3, 0x0, 0x400a148)
/go/src/github.com/influxdata/influxdb/cmd/influxd/run/command.go:149 +0x7e4
main.(*Main).Run(0x429df8c, 0x400a148, 0x2, 0x3, 0x2b5a4e0, 0x4066070)
/go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:81 +0x104
main.main()
/go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:45 +0x140
goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:147 +0x130
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x14
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x30
goroutine 20 [select]:
github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view.(*worker).start(0x42ca780)
/go/src/github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view/worker.go:154 +0xb0
created by github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view.init.0
/go/src/github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view/worker.go:32 +0x48
goroutine 21 [IO wait]:
internal/poll.runtime_pollWait(0xa6c0b130, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:184 +0x44
internal/poll.(*pollDesc).wait(0x412ac44, 0x72, 0x0, 0x0, 0xfd0405)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0x412ac30, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1a8
net.(*netFD).accept(0x412ac30, 0x0, 0xa7801, 0x0)
/usr/local/go/src/net/fd_unix.go:238 +0x20
net.(*TCPListener).accept(0x4363a90, 0x4338f40, 0x40000000, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x20
net.(*TCPListener).Accept(0x4363a90, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock.go:261 +0x3c
github.com/influxdata/influxdb/tcp.(*Mux).Serve(0x4338f40, 0x1eb0e70, 0x4363a90, 0x4363a90, 0x0)
/go/src/github.com/influxdata/influxdb/tcp/mux.go:75 +0x64
created by github.com/influxdata/influxdb/cmd/influxd/run.(*Server).Open
/go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:395 +0x1f0
ts=2023-10-23T21:37:37.154508Z lvl=info msg="InfluxDB starting" log_id=0l3cOmWW000 version=1.8.5 branch=1.8 commit=3d16c6318cf5
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the startup path shown in the trace: tsdb/engine/tsm1/cache.go, especially Cache.WriteMulti and CacheLoader.Load, then follow engine.go reloadCache and tsdb/store.go loadShards. Reproduce startup with the reported InfluxDB 1.8.5 Docker, Raspberry Pi 4, and 32-bit Raspbian setup; done means the existing data can load without the container restarting from an out-of-memory failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- database, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100