filecoin-project / filecoin-project/boost

panic: runtime error: index out of range

Open
#1,713 1 comment 0 reactions 0 assignees View on GitHub
kind/bug need/triage
Dominant language
Go
Stars
121
Forks
79
Avg merge
10d 14h
Merged PRs (30d)
1

Description

### Checklist

- [X] This is **not** a question or a support request. If you have any boost related questions, please ask in the [discussion forum](https://github.com/filecoin-project/boost/discussions).
- [X] This is **not** a new feature request. If it is, please file a [feature request](https://github.com/filecoin-project/boost/issues/new?assignees=&labels=need%2Ftriage%2Ckind%2Ffeature&template=feature_request.yml) instead.
- [X] This is **not** an enhancement request. If it is, please file a [improvement suggestion](https://github.com/filecoin-project/boost/issues/new?assignees=&labels=need%2Ftriage%2Ckind%2Fenhancement&template=enhancement.yml) instead.
- [X] I **have** searched on the [issue tracker](https://github.com/filecoin-project/boost/issues) and the [discussion forum](https://github.com/filecoin-project/boost/discussions), and there is no existing related issue or discussion.
- [X] I am running the [`Latest release`](https://github.com/filecoin-project/boost/releases), or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
- [X] I did not make any code changes to boost.

### Boost component

- [X] boost daemon - storage providers
- [ ] boost client
- [ ] boost UI
- [ ] boost data-transfer
- [ ] boost index-provider
- [ ] Other

### Boost Version

```text
boostd version 1.7.4+git.52cb7cd
```

### Describe the Bug

For some reason, boostd often panics. There is no sealing. Do you suspect it may be caused by retrieving data?

### Logging Information

```text
....
{"level":"debug","ts":"2023-09-23T19:31:57.667+0800","logger":"dagstore","caller":"dagstore@v0.7.0/dagstore_control.go:341","msg":"finished processing task","op":"OpShardAcquire","shard":"baga6ea4seaqe
crifbmvwi7rpvcmpxy3mygh76x2hbq2dlqqj6pllafz6ed246aq","prev_state":"ShardStateAvailable","curr_state":"ShardStateServing","error":null}
{"level":"debug","ts":"2023-09-23T19:31:57.667+0800","logger":"dagstore","caller":"dagstore/wrapper.go:169","msg":"trace","shard-key":"baga6ea4seaqecrifbmvwi7rpvcmpxy3mygh76x2hbq2dlqqj6pllafz6ed246aq",
"op-type":"OpShardAcquire","after":"ShardStateServing"}
{"level":"debug","ts":"2023-09-23T19:31:57.719+0800","logger":"dagstore","caller":"dagstore@v0.7.0/dagstore_async.go:53","msg":"acquire: successfully fetched from mount upgrader","shard":"baga6ea4seaqe
crifbmvwi7rpvcmpxy3mygh76x2hbq2dlqqj6pllafz6ed246aq"}
{"level":"debug","ts":"2023-09-23T19:31:57.719+0800","logger":"dagstore","caller":"dagstore@v0.7.0/dagstore_async.go:87","msg":"acquire: successful; returning accessor","shard":"baga6ea4seaqecrifbmvwi7
rpvcmpxy3mygh76x2hbq2dlqqj6pllafz6ed246aq"}
{"level":"error","ts":"2023-09-23T19:31:57.802+0800","logger":"dt-impl","caller":"impl/receiver.go:135","msg":"received error message on data transfer: stream reset"}
{"level":"debug","ts":"2023-09-23T19:31:58.477+0800","logger":"dagstore","caller":"dagstore@v0.7.0/dagstore_async.go:125","msg":"initialize: finished generating index for shard","shard":"baga6ea4seaqap
zwqmwi6eg7k6zsewhnfbtjowrcnpspmt2bzwm3ytiw37x43caq"}
panic: runtime error: index out of range [131043] with length 130048

goroutine 75542 [running]:
bufio.(*Reader).Read(0xf3d3654540, {0xf3dda00000, 0x1fc00, 0x7a83ea?})
/usr/local/go/src/bufio/bufio.go:228 +0x31f
bufio.(*Reader).Read(0xf3d36545a0, {0xc01429ab68, 0x1, 0x1?})
/usr/local/go/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x68740a0, 0xf3d36545a0}, {0xc01429ab68, 0x1, 0x1}, 0x1)
/usr/local/go/src/io/io.go:332 +0x9a
io.ReadFull(...)
/usr/local/go/src/io/io.go:351
github.com/filecoin-project/lotus/storage/sealer.(*pieceReader).readSeqReader(0xf38c32d170, {0xc01429ab68, 0x1, 0x1})
/root/go/pkg/mod/github.com/filecoin-project/lotus@v1.23.2-0.20230622154405-168d022018ce/storage/sealer/piece_reader.go:200 +0xaf5
github.com/filecoin-project/lotus/storage/sealer.(*pieceReader).Read(0xf38c32d170, {0xc01429ab68, 0x1, 0x1})
/root/go/pkg/mod/github.com/filecoin-project/lotus@v1.23.2-0.20230622154405-168d022018ce/storage/sealer/piece_reader.go:118 +0x151
io.ReadAtLeast({0x687abc0, 0xc01429ab58}, {0xc01429ab68, 0x1, 0x1}, 0x1)
/usr/local/go/src/io/io.go:332 +0x9a
io.ReadFull(...)
/usr/local/go/src/io/io.go:351
github.com/ipld/go-car/v2/internal/io.(*readerPlusByte).ReadByte(0xc01429ab58)
/root/go/pkg/mod/github.com/ipld/go-car/v2@v2.10.1/internal/io/converter.go:87 +0x39
github.com/multiformats/go-varint.ReadUvarint({0x687aba0, 0xc01429ab58})
/root/go/pkg/mod/github.com/multiformats/go-varint@v0.0.7/varint.go:80 +0x5a
github.com/ipld/go-car/v2/internal/carv1/util.LdReadSize({0x7fd9c45d2410?, 0xf38c32d170}, 0x0, 0x2000000)
/root/go/pkg/mod/github.com/ipld/go-car/v2@v2.10.1/internal/carv1/util/util.go:69 +0x96
github.com/ipld/go-car/v2/internal/carv1/util.LdRead({0x7fd9c45d2410, 0xf38c32d170}, 0x0?, 0xc3c9c1aa28?)
/root/go/pkg/mod/github.com/ipld/go-car/v2@v2.10.1/internal/carv1/util/util.go:87 +0x27
github.com/ipld/go-car/v2/internal/carv1.ReadHeader({0x7fd9c45d2410?, 0xf38c32d170?}, 0x401?)
/root/go/pkg/mod/github.com/ipld/go-car/v2@v2.10.1/internal/carv1/car.go:52 +0x32
github.com/ipld/go-car/v2.ReadVersion({0x7fd9c45d2410, 0xf38c32d170}, {0xc3c9c1ac98?, 0x4ea1220?, 0xc4d6807f38?})
/root/go/pkg/mod/github.com/ipld/go-car/v2@v2.10.1/reader.go:363 +0x90
github.com/ipld/go-car/v2.ReadOrGenerateIndex({0x7fd9c45d23e8?, 0xf38c32d170}, {0xc06a5a9c98, 0x2, 0x2})
.....
```

### Repo Steps

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the panic trace, especially impl/receiver.go:135, lotus storage/sealer/piece_reader.go, and go-car v2 reader.go and CAR utilities. Reproduce the boostd 1.7.4 storage-provider failure around dagstore index generation and determine the input or read path causing the out-of-range panic; done means the panic is prevented or handled and the relevant regression is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.