influxdata / influxdata/influxdb
influx_inspect buildtsi -compact-series-file -> unexpected fault address 0xa4ba5000
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Hi,
I upgraded my influxdb:1.7.10 docker container to 1.8.0 and migrated to INFLUXDB_DATA_INDEX_VERSION=tsi1, which was successful.
I'm not sure if the upgrade has something to do with the segfault of influx_inspect so I'm adding my upgrade steps as well.
Afterwards I wanted to use "influx_inspect buildtsi -compact-series-file ..." but it crashes with a "fatal error: fault"
I tried starting the container with those data files but the container just loops and will not start.
__Environment info:__
My docker environment runs on Raspberry Pi 4 with the data stored on an external SSD.
```
root@andromeda:~ # uname -srm
Linux 4.19.97-v7l+ armv7l
```
When influxdb container is running there is enough free memory available.
```
root@andromeda:~ # free -m
total used free shared buff/cache available
Mem: 3906 1055 95 34 2755 2695
Swap: 8291 229 8062
```
__Steps to reproduce:__
__Successful upgrade 1.7.10 to 1.8.0 steps__
I read https://docs.influxdata.com/influxdb/v1.8/administration/upgrading/ and decided to upgrade my container from 1.7.10 to 1.8.0 and also use tsi1 index.
My starting docker-compose definition was:
```
influxdb:
container_name: mnetcontrol-influxdb
hostname: mnetcontrol-influxdb
restart: "always"
expose:
- "8086"
image: influxdb:1.7.10
volumes:
- ./influxdb/var/lib/influxdb:/var/lib/influxdb
environment:
TZ: "Europe/Berlin"
INFLUXDB_HTTP_AUTH_ENABLED: "true"
INFLUXDB_ADMIN_USER: "admin"
INFLUXDB_ADMIN_PASSWORD: "${INFLUXDB_ADMIN_PASSWORD}"
networks:
- mnetcontrol
```
I had no index files as mentioned in 3c) of the upgrade instructions
So my understanding was that I didn't need to prepare the data somehow because I had no index files.
```
root@mnetcontrol-influxdb:/# find /var/lib/influxdb/ | grep -i index
root@mnetcontrol-influxdb:/#
```
My config file/container environment variables contained no index-version setting as mentioned in 3a)
```
root@mnetcontrol-influxdb:/# cat /etc/influxdb/influxdb.conf
[meta]
dir = "/var/lib/influxdb/meta"
[data]
dir = "/var/lib/influxdb/data"
engine = "tsm1"
wal-dir = "/var/lib/influxdb/wal"
root@mnetcontrol-influxdb:/#
```
I then stopped the container and backuped the data files
```
root@andromeda:/opt/docker/mnetcontrol # docker container stop mnetcontrol-influxdb
root@andromeda:/opt/docker/mnetcontrol # cp -a influxdb backups/influxdb_2020-04-24_1.7.10
```
Started a temporary container to build the index
```
root@andromeda:/opt/docker/mnetcontrol # docker run --rm \
-it \
-e INFLUXDB_DATA_INDEX_VERSION=tsi1 \
--entrypoint /bin/bash \
-v /opt/docker/mnetcontrol/influxdb/var/lib/influxdb:/var/lib/influxdb \
-p 8086 \
influxdb:1.8.0
Unable to find image 'influxdb:1.8.0' locally
1.8.0: Pulling from library/influxdb
4168e46f368a: Pull complete
...
```
Ran inside the container the "influx_inspect buildtsi ..." which ran fine reported only "info" messages and as far as I can tell reported no errors.
```
root@159f1033c05c:/# influx_inspect buildtsi -waldir /var/lib/influxdb/wal -datadir /var/lib/influxdb/data
```
Modified docker-compose definition from above with the following values
```
root@andromeda:/opt/docker/mnetcontrol # grep -A10 influxdb docker-compose.yml
...
influxdb:
#image: influxdb:1.7.10 # since 2020-04-04
image: influxdb:1.8.0 # since 2020-04-24
environment:
# since version 1.8.0 index tsi1 is recommend
INFLUXDB_DATA_INDEX_VERSION: "tsi1"
...
```
Started container
```
root@andromeda:/opt/docker/mnetcontrol# docker-compose up -d influxdb
```
Checked my grafana dashboards afterwards and the data from last year for all sensors looked fine.
__Failed steps to compact-series in 1.8.0__
Stopped the 1.8.0 container
```
root@andromeda:/opt/docker/mnetcontrol # docker container stop mnetcontrol-influxdb
```
Created a temporary container with influxdb and access to the data files
```
root@andromeda:/opt/docker/mnetcontrol # docker run --rm \
> -it \
> -e INFLUXDB_DATA_INDEX_VERSION=tsi1 \
> --entrypoint /bin/bash \
> -v /opt/docker/mnetcontrol/influxdb/var/lib/influxdb:/var/lib/influxdb \
> -p 8086 \
> influxdb:1.8.0
```
Ran the compact-series command as described here: https://docs.influxdata.com/influxdb/v1.8/administration/compact-series-file/
And you can see the result below
```
root@134ce7be7115:/# influx_inspect buildtsi -compact-series-file -waldir /var/lib/influxdb/wal -datadir /var/lib/influxdb/data
You are currently running as root. This will build your
index files with root ownership and will be inaccessible
if you run influxd as a non-root user. You should run
buildtsi as the same user you are running influxd.
Are you sure you want to continue? (y/N): y
processing partition for "/var/lib/influxdb/data/_internal/_series/01"
processing partition for "/var/lib/influxdb/data/_internal/_series/00"
processing partition for "/var/lib/influxdb/data/_internal/_series/02"
processing partition for "/var/lib/influxdb/data/_internal/_series/03"
processing segment "/var/lib/influxdb/data/_internal/_series/01/0000" 0
processing segment "/var/lib/influxdb/data/_internal/_series/00/0000" 0
processing segment "/var/lib/influxdb/data/_internal/_series/02/0000" 0
processing segment "/var/lib/influxdb/data/_internal/_series/03/0000" 0
renaming new segment "/var/lib/influxdb/data/_internal/_series/00/0000.tmp" to "/var/lib/influxdb/data/_internal/_series/00/0000"
removing index file /var/lib/influxdb/data/_internal/_series/00/index
processing partition for "/var/lib/influxdb/data/_internal/_series/04"
renaming new segment "/var/lib/influxdb/data/_internal/_series/01/0000.tmp" to "/var/lib/influxdb/data/_internal/_series/01/0000"
renaming new segment "/var/lib/influxdb/data/_internal/_series/03/0000.tmp" to "/var/lib/influxdb/data/_internal/_series/03/0000"
removing index file /var/lib/influxdb/data/_internal/_series/01/index
processing partition for "/var/lib/influxdb/data/_internal/_series/05"
removing index file /var/lib/influxdb/data/_internal/_series/03/index
processing partition for "/var/lib/influxdb/data/_internal/_series/06"
renaming new segment "/var/lib/influxdb/data/_internal/_series/02/0000.tmp" to "/var/lib/influxdb/data/_internal/_series/02/0000"
removing index file /var/lib/influxdb/data/_internal/_series/02/index
processing partition for "/var/lib/influxdb/data/_internal/_series/07"
processing segment "/var/lib/influxdb/data/_internal/_series/04/0000" 0
processing segment "/var/lib/influxdb/data/_internal/_series/06/0000" 0
processing segment "/var/lib/influxdb/data/_internal/_series/05/0000" 0
renaming new segment "/var/lib/influxdb/data/_internal/_series/06/0000.tmp" to "/var/lib/influxdb/data/_internal/_series/06/0000"
processing segment "/var/lib/influxdb/data/_internal/_series/07/0000" 0
removing index file /var/lib/influxdb/data/_internal/_series/06/index
renaming new segment "/var/lib/influxdb/data/_internal/_series/05/0000.tmp" to "/var/lib/influxdb/data/_internal/_series/05/0000"
renaming new segment "/var/lib/influxdb/data/_internal/_series/04/0000.tmp" to "/var/lib/influxdb/data/_internal/_series/04/0000"
removing index file /var/lib/influxdb/data/_internal/_series/05/index
removing index file /var/lib/influxdb/data/_internal/_series/04/index
renaming new segment "/var/lib/influxdb/data/_internal/_series/07/0000.tmp" to "/var/lib/influxdb/data/_internal/_series/07/0000"
removing index file /var/lib/influxdb/data/_internal/_series/07/index
unexpected fault address 0xa4ba5000
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0xa4ba5000 pc=0x48814c]
goroutine 1 [running]:
runtime.throw(0x6ae788, 0x5)
/usr/local/go/src/runtime/panic.go:774 +0x5c fp=0x108aa84 sp=0x108aa70 pc=0x3f948
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:391 +0x378 fp=0x108aa9c sp=0x108aa84 pc=0x55220
github.com/influxdata/influxdb/tsdb.ReadSeriesEntry(0xa4ba5000, 0x3f7000, 0x3f7000, 0x8f12, 0x0, 0xa4ba4f1b, 0xe5, 0x3f70e5, 0x0, 0x0, ...)
/go/src/github.com/influxdata/influxdb/tsdb/series_segment.go:417 +0x20 fp=0x108aad4 sp=0x108aaa0 pc=0x48814c
github.com/influxdata/influxdb/tsdb.(*SeriesSegment).ForEachEntry(0x10742a0, 0x108ab28, 0x2, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/series_segment.go:244 +0x64 fp=0x108ab0c sp=0x108aad4 pc=0x4877c8
github.com/influxdata/influxdb/tsdb.(*SeriesSegment).MaxSeriesID(0x10742a0, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/series_segment.go:232 +0x50 fp=0x108ab30 sp=0x108ab0c pc=0x48773c
github.com/influxdata/influxdb/tsdb.(*SeriesPartition).openSegments(0x1076070, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:124 +0x2a4 fp=0x108aba4 sp=0x108ab30 pc=0x4838b4
github.com/influxdata/influxdb/tsdb.(*SeriesPartition).Open.func1(0x1076070, 0x2b, 0x1ff)
/go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:78 +0x1c fp=0x108abd8 sp=0x108aba4 pc=0x48c3cc
github.com/influxdata/influxdb/tsdb.(*SeriesPartition).Open(0x1076070, 0x1074240, 0x1)
/go/src/github.com/influxdata/influxdb/tsdb/series_partition.go:95 +0x90 fp=0x108ac00 sp=0x108abd8 pc=0x483568
github.com/influxdata/influxdb/tsdb.(*SeriesFile).Open(0x1238380, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/tsdb/series_file.go:91 +0x260 fp=0x108adb8 sp=0x108ac00 pc=0x47ecf8
github.com/influxdata/influxdb/cmd/influx_inspect/buildtsi.(*Command).compactDatabaseSeriesFile(0x1026d20, 0x10f6097, 0x9, 0x10f60a0, 0x20, 0x0, 0x0)
/go/src/github.com/influxdata/influxdb/cmd/influx_inspect/buildtsi/buildtsi.go:168 +0x1ec fp=0x108ae38 sp=0x108adb8 pc=0x587840
github.com/influxdata/influxdb/cmd/influx_inspect/buildtsi.(*Command).run(0x1026d20, 0xbeab1eff, 0x16, 0xbeab1ee0, 0x15, 0x0, 0x7)
/go/src/github.com/influxdata/influxdb/cmd/influx_inspect/buildtsi/buildtsi.go:120 +0x1d8 fp=0x108aec8 sp=0x108ae38 pc=0x5871a8
github.com/influxdata/influxdb/cmd/influx_inspect/buildtsi.(*Command).Run(0x1026d20, 0x1024150, 0x5, 0x6, 0x8, 0x1024150)
/go/src/github.com/influxdata/influxdb/cmd/influx_inspect/buildtsi/buildtsi.go:81 +0x4b0 fp=0x108aef8 sp=0x108aec8 pc=0x586edc
main.(*Main).Run(0x108af88, 0x1024150, 0x5, 0x6, 0x0, 0x14f01)
/go/src/github.com/influxdata/influxdb/cmd/influx_inspect/main.go:93 +0xc40 fp=0x108af58 sp=0x108aef8 pc=0x5ae560
main.main()
/go/src/github.com/influxdata/influxdb/cmd/influx_inspect/main.go:28 +0x110 fp=0x108afa4 sp=0x108af58 pc=0x5ad84c
runtime.main()
/usr/local/go/src/runtime/proc.go:203 +0x208 fp=0x108afe4 sp=0x108afa4 pc=0x41c28
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:868 +0x4 fp=0x108afe4 sp=0x108afe4 pc=0x6dd38
```
Contributor guide
Assessment
This issue has not been assessed yet.