influxdata / influxdata/influxdb
write points fail and report [shard 90] short buffer?
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. write point use java client
2. client report
```
com.influxdb.exceptions.InternalServerErrorException: unexpected error writing points to database: [shard 90] short buffer
at com.influxdb.internal.AbstractRestClient.responseToError(AbstractRestClient.java:118) ~[influxdb-client-core-3.4.0.jar!/:3.4.0]
at com.influxdb.internal.AbstractRestClient.execute(AbstractRestClient.java:81) ~[influxdb-client-core-3.4.0.jar!/:3.4.0]
at com.influxdb.client.internal.AbstractWriteBlockingClient.write(AbstractWriteBlockingClient.java:85) ~[influxdb-client-java-3.4.0.jar!/:3.4.0]
at com.influxdb.client.internal.WriteApiBlockingImpl.lambda$writePoints$1(WriteApiBlockingImpl.java:137) ~[influxdb-client-java-3.4.0.jar!/:3.4.0]
```
3. influxdb logs error too ,and only when write some data trigger this error
__Environment info:__
docker
influx 2.1.1
__Logs:__
Include snippet of errors in log.
```
2021-12-01T06:25:36.585734Z info Reading file {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "service": "cacheloader", "path": "/var/lib/influxdb2/engine/wal/cb8da52d13f762fb/autogen/96/_00001.wal", "size": 5712}
2021-12-01T06:25:36.591886Z info Opened file {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "service": "filestore", "path": "/var/lib/influxdb2/engine/data/cb8da52d13f762fb/autogen/94/000000001-000000001.tsm", "id": 0, "duration": "7.343ms"}
2021-12-01T06:25:36.611728Z info Opened file {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "service": "filestore", "path": "/var/lib/influxdb2/engine/data/cb8da52d13f762fb/autogen/89/000000003-000000002.tsm", "id": 0, "duration": "0.596ms"}
2021-12-01T06:25:43.419126Z error Failed to open shard {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "store", "op_name": "tsdb_open", "db_shard_id": 90, "error": "[shard 90] short buffer"}
2021-12-01T06:25:44.798705Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.808490Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.810225Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.851809Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.862204Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
```
```
2021-12-01T07:00:56.567977Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:05:56.869878Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:03.029356Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:10.678191Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:10.694187Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:19.697167Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:19.706261Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:28.065021Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:28.074078Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:36.696444Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:36.705558Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:42.150872Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:42.159537Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:48.998667Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:49.007475Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:56.541653Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:56.550789Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:04.432700Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:04.441095Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:13.482655Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:13.492085Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:21.475071Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:21.483124Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:29.106772Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
```
```
2021-12-02T02:55:50.189347Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T02:56:46.791163Z info Retention policy deletion check (start) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "start"}
2021-12-02T02:56:46.791346Z info Retention policy deletion check (end) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "end", "op_elapsed": "0.206ms"}
2021-12-02T02:58:51.197395Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T03:00:50.281280Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T03:03:44.792549Z info Cache snapshot (start) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:03:44.824314Z info Snapshot for path written {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/d985752f6c791b5b/autogen/20", "duration": "31.791ms"}
2021-12-02T03:03:44.824357Z info Cache snapshot (end) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "31.833ms"}
2021-12-02T03:03:49.558056Z info Cache snapshot (start) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:03:49.588914Z info Snapshot for path written {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/592474454cfee085/autogen/99", "duration": "30.889ms"}
2021-12-02T03:03:49.588946Z info Cache snapshot (end) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "30.919ms"}
2021-12-02T03:03:53.616409Z info Cache snapshot (start) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:03:53.648065Z info Snapshot for path written {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/e801bb6a29638f3e/autogen/100", "duration": "31.674ms"}
2021-12-02T03:03:53.648100Z info Cache snapshot (end) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "31.709ms"}
2021-12-02T03:04:25.317616Z info Unauthorized {"log_id": "0Y9AZ_e0000", "error": "token required"}
2021-12-02T03:04:55.852016Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 776506 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc00f43eed0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc0587a6540}, 0xc0617efbd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc0587a6540}, 0xc00f43e7b0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01c044180, 0x61a837d7)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc00fa53860, {0x40a6b18, 0xc0587a6540}, {0x406f510, 0xc01c044180}, 0xc016b2c0b0)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc00fa53860, {0x40a6b18, 0xc0587a6540})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc00fa53860, {0x40a6b18, 0xc0587a6540})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc00fa53860})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:04:57.479246Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 776589 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01d85f710})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc015cda100}, 0xc01758fbd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc015cda100}, 0xc01d85f020)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01d839380, 0x61a837d9)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc00b639a00, {0x40a6b18, 0xc015cda100}, {0x406f510, 0xc01d839380}, 0xc03c8df0e0)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc00b639a00, {0x40a6b18, 0xc015cda100})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc00b639a00, {0x40a6b18, 0xc015cda100})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc00b639a00})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:04:59.934067Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 776621 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01e71b800})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc0138f7340}, 0xc03d883bd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc0138f7340}, 0xc01e71b110)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01e70d100, 0x61a837db)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc0181f2d00, {0x40a6b18, 0xc0138f7340}, {0x406f510, 0xc01e70d100}, 0xc07acbf840)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc0181f2d00, {0x40a6b18, 0xc0138f7340})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc0181f2d00, {0x40a6b18, 0xc0138f7340})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc0181f2d00})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:05:50.338055Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T03:08:09.183024Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-02T03:08:09.183373Z info Reindexing TSM data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 101}
2021-12-02T03:08:09.183392Z info Reindexing WAL data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 101}
2021-12-02T03:09:00.159441Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-02T03:09:00.159977Z info Reindexing TSM data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 102}
2021-12-02T03:09:00.159995Z info Reindexing WAL data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 102}
2021-12-02T03:09:54.759458Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-02T03:09:54.759771Z info Reindexing TSM data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 103}
2021-12-02T03:09:54.759784Z info Reindexing WAL data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 103}
2021-12-02T03:14:03.878068Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 903477 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc00b13f740})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc01c242cc0}, 0xc028b75bd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc01c242cc0}, 0xc00b13f5f0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc00257e380, 0x61a839fb)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc031c8c410, {0x40a6b18, 0xc01c242cc0}, {0x406f510, 0xc00257e380}, 0xc0d402d520)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc031c8c410, {0x40a6b18, 0xc01c242cc0})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc031c8c410, {0x40a6b18, 0xc01c242cc0})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc031c8c410})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:14:07.591332Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 903622 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01c3ea150})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc01cf75240}, 0xc016bf8bd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc01cf75240}, 0xc01c3ea000)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01b72c400, 0x61a839ff)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc031c8cc30, {0x40a6b18, 0xc01cf75240}, {0x406f510, 0xc01b72c400}, 0xc0116b95b0)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc031c8cc30, {0x40a6b18, 0xc01cf75240})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc031c8cc30, {0x40a6b18, 0xc01cf75240})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc031c8cc30})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:14:14.450359Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 904050 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01d8e8db0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc01daf9340}, 0xc002a4abd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc01daf9340}, 0xc01d8e8c60)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01d496800, 0x61a83a06)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc0173ff040, {0x40a6b18, 0xc01daf9340}, {0x406f510, 0xc01d496800}, 0xc02bd34880)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc0173ff040, {0x40a6b18, 0xc01daf9340})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc0173ff040, {0x40a6b18, 0xc01daf9340})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc0173ff040})
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:15:51.054091Z info Cache snapshot (start) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:15:51.057282Z info Snapshot for path written {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/cb8da52d13f762fb/autogen/98", "duration": "3.215ms"}
2021-12-02T03:15:51.057308Z info Cache snapshot (end) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "3.238ms"}
2021-12-02T03:17:16.205704Z warn internal error not returned to client {"log_id": "0Y9AZ_e0000", "handler": "error_logger", "error": "context canceled"}
2021-12-02T03:26:46.791140Z info Retention policy deletion check (start) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "start"}
2021-12-02T03:26:46.791292Z info Retention policy deletion check (end) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "end", "op_elapsed": "0.164ms"}
```
Contributor guide
Research direction
Start with v1/services/storage/store.go around measurementFields and TagValues, then trace the shard-opening path associated with the logged [shard 90] short buffer error. Reproduce the write through the Java client against the Docker-based InfluxDB 2.1.1 environment; done means writes no longer fail for the affected shard.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100