influxdata / influxdata/influxdb

write points fail and report [shard 90] short buffer?

Open
#22,951 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. write point use java client
2. client report
```
com.influxdb.exceptions.InternalServerErrorException: unexpected error writing points to database: [shard 90] short buffer
at com.influxdb.internal.AbstractRestClient.responseToError(AbstractRestClient.java:118) ~[influxdb-client-core-3.4.0.jar!/:3.4.0]
at com.influxdb.internal.AbstractRestClient.execute(AbstractRestClient.java:81) ~[influxdb-client-core-3.4.0.jar!/:3.4.0]
at com.influxdb.client.internal.AbstractWriteBlockingClient.write(AbstractWriteBlockingClient.java:85) ~[influxdb-client-java-3.4.0.jar!/:3.4.0]
at com.influxdb.client.internal.WriteApiBlockingImpl.lambda$writePoints$1(WriteApiBlockingImpl.java:137) ~[influxdb-client-java-3.4.0.jar!/:3.4.0]
```

3. influxdb logs error too ,and only when write some data trigger this error

__Environment info:__

docker
influx 2.1.1

__Logs:__
Include snippet of errors in log.

```
2021-12-01T06:25:36.585734Z info Reading file {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "service": "cacheloader", "path": "/var/lib/influxdb2/engine/wal/cb8da52d13f762fb/autogen/96/_00001.wal", "size": 5712}
2021-12-01T06:25:36.591886Z info Opened file {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "service": "filestore", "path": "/var/lib/influxdb2/engine/data/cb8da52d13f762fb/autogen/94/000000001-000000001.tsm", "id": 0, "duration": "7.343ms"}
2021-12-01T06:25:36.611728Z info Opened file {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "service": "filestore", "path": "/var/lib/influxdb2/engine/data/cb8da52d13f762fb/autogen/89/000000003-000000002.tsm", "id": 0, "duration": "0.596ms"}
2021-12-01T06:25:43.419126Z error Failed to open shard {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "store", "op_name": "tsdb_open", "db_shard_id": 90, "error": "[shard 90] short buffer"}
2021-12-01T06:25:44.798705Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.808490Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.810225Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.851809Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.862204Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
```

```
2021-12-01T07:00:56.567977Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:05:56.869878Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:03.029356Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:10.678191Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:10.694187Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:19.697167Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:19.706261Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:28.065021Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:28.074078Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:36.696444Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:36.705558Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:42.150872Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:42.159537Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:48.998667Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:49.007475Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:56.541653Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:56.550789Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:04.432700Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:04.441095Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:13.482655Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:13.492085Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:21.475071Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:21.483124Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:29.106772Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
```

```
2021-12-02T02:55:50.189347Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T02:56:46.791163Z info Retention policy deletion check (start) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "start"}
2021-12-02T02:56:46.791346Z info Retention policy deletion check (end) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "end", "op_elapsed": "0.206ms"}
2021-12-02T02:58:51.197395Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T03:00:50.281280Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T03:03:44.792549Z info Cache snapshot (start) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:03:44.824314Z info Snapshot for path written {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/d985752f6c791b5b/autogen/20", "duration": "31.791ms"}
2021-12-02T03:03:44.824357Z info Cache snapshot (end) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "31.833ms"}
2021-12-02T03:03:49.558056Z info Cache snapshot (start) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:03:49.588914Z info Snapshot for path written {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/592474454cfee085/autogen/99", "duration": "30.889ms"}
2021-12-02T03:03:49.588946Z info Cache snapshot (end) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "30.919ms"}
2021-12-02T03:03:53.616409Z info Cache snapshot (start) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:03:53.648065Z info Snapshot for path written {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/e801bb6a29638f3e/autogen/100", "duration": "31.674ms"}
2021-12-02T03:03:53.648100Z info Cache snapshot (end) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "31.709ms"}
2021-12-02T03:04:25.317616Z info Unauthorized {"log_id": "0Y9AZ_e0000", "error": "token required"}
2021-12-02T03:04:55.852016Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 776506 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc00f43eed0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc0587a6540}, 0xc0617efbd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc0587a6540}, 0xc00f43e7b0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01c044180, 0x61a837d7)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc00fa53860, {0x40a6b18, 0xc0587a6540}, {0x406f510, 0xc01c044180}, 0xc016b2c0b0)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc00fa53860, {0x40a6b18, 0xc0587a6540})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc00fa53860, {0x40a6b18, 0xc0587a6540})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc00fa53860})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:04:57.479246Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 776589 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01d85f710})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc015cda100}, 0xc01758fbd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc015cda100}, 0xc01d85f020)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01d839380, 0x61a837d9)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc00b639a00, {0x40a6b18, 0xc015cda100}, {0x406f510, 0xc01d839380}, 0xc03c8df0e0)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc00b639a00, {0x40a6b18, 0xc015cda100})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc00b639a00, {0x40a6b18, 0xc015cda100})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc00b639a00})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:04:59.934067Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 776621 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01e71b800})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc0138f7340}, 0xc03d883bd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc0138f7340}, 0xc01e71b110)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01e70d100, 0x61a837db)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc0181f2d00, {0x40a6b18, 0xc0138f7340}, {0x406f510, 0xc01e70d100}, 0xc07acbf840)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc0181f2d00, {0x40a6b18, 0xc0138f7340})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc0181f2d00, {0x40a6b18, 0xc0138f7340})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc0181f2d00})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:05:50.338055Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T03:08:09.183024Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-02T03:08:09.183373Z info Reindexing TSM data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 101}
2021-12-02T03:08:09.183392Z info Reindexing WAL data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 101}
2021-12-02T03:09:00.159441Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-02T03:09:00.159977Z info Reindexing TSM data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 102}
2021-12-02T03:09:00.159995Z info Reindexing WAL data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 102}
2021-12-02T03:09:54.759458Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-02T03:09:54.759771Z info Reindexing TSM data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 103}
2021-12-02T03:09:54.759784Z info Reindexing WAL data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 103}
2021-12-02T03:14:03.878068Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 903477 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc00b13f740})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc01c242cc0}, 0xc028b75bd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc01c242cc0}, 0xc00b13f5f0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc00257e380, 0x61a839fb)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc031c8c410, {0x40a6b18, 0xc01c242cc0}, {0x406f510, 0xc00257e380}, 0xc0d402d520)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc031c8c410, {0x40a6b18, 0xc01c242cc0})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc031c8c410, {0x40a6b18, 0xc01c242cc0})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc031c8c410})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:14:07.591332Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 903622 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01c3ea150})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc01cf75240}, 0xc016bf8bd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc01cf75240}, 0xc01c3ea000)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01b72c400, 0x61a839ff)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc031c8cc30, {0x40a6b18, 0xc01cf75240}, {0x406f510, 0xc01b72c400}, 0xc0116b95b0)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc031c8cc30, {0x40a6b18, 0xc01cf75240})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc031c8cc30, {0x40a6b18, 0xc01cf75240})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc031c8cc30})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:14:14.450359Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 904050 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01d8e8db0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc01daf9340}, 0xc002a4abd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc01daf9340}, 0xc01d8e8c60)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01d496800, 0x61a83a06)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc0173ff040, {0x40a6b18, 0xc01daf9340}, {0x406f510, 0xc01d496800}, 0xc02bd34880)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc0173ff040, {0x40a6b18, 0xc01daf9340})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc0173ff040, {0x40a6b18, 0xc01daf9340})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc0173ff040})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:15:51.054091Z info Cache snapshot (start) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:15:51.057282Z info Snapshot for path written {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/cb8da52d13f762fb/autogen/98", "duration": "3.215ms"}
2021-12-02T03:15:51.057308Z info Cache snapshot (end) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "3.238ms"}
2021-12-02T03:17:16.205704Z warn internal error not returned to client {"log_id": "0Y9AZ_e0000", "handler": "error_logger", "error": "context canceled"}
2021-12-02T03:26:46.791140Z info Retention policy deletion check (start) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "start"}
2021-12-02T03:26:46.791292Z info Retention policy deletion check (end) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "end", "op_elapsed": "0.164ms"}
```

Contributor guide

Open the contributing guide

Research direction

Start with v1/services/storage/store.go around measurementFields and TagValues, then trace the shard-opening path associated with the logged [shard 90] short buffer error. Reproduce the write through the Java client against the Docker-based InfluxDB 2.1.1 environment; done means writes no longer fail for the affected shard.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, java
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.