filecoin-project / filecoin-project/boost
[Support Ticket]: Boost stops accepting deals, performing retrievals, deals are stuck in 'Announcing' state, 'Piece Doctor' page does not load.
- Dominant language
- Go
- Stars
- 121
- Forks
- 79
- Avg merge
- 10d 14h
- Merged PRs (30d)
- 1
Description
### Boost component
- [X] boost daemon - storage providers
- [ ] boost client
- [X] boost UI
- [ ] boost data-transfer
- [ ] boost index-provider
- [ ] booster-http
- [ ] booster-bitswap
- [X] LID Database - Yugabyte/LevelDB
- [X] boostd-data
- [ ] Other
### Boost Version
```text
boost version 2.1.1+mainnet+git.e9d18ac
boostd version 2.1.1+mainnet+git.e9d18ac
boostd-data version 1.5.1+mainnet+git.e9d18ac
booster-bitswap version 2.1.1+mainnet+git.e9d18ac
```
### Describe the problem
After some time of more or less successful work (it can be several hours or a day or two), something goes wrong - `boostd` stops accepting deals, performing retrievals, deals are stuck in 'Announcing' state, 'Piece Doctor' page does not load.
>Yugabyte setup details
YugabyteDB v2.18.2.1 (replication factor 3, 3 nodes)
YugabyteDB nodes are running on Lenovo P620 workstations. HW specs of 3 hosts:
Host | OS | YB NVME | FS | CPU | RAM | Role
-- | -- | -- | -- | -- | -- | --
len4 | Ubuntu 22.04.3 LTS | WD Black SN750 2TB WDS200T3X0C-00SJG0 | XFS | AMD Ryzen Threadripper PRO 3955WX 16-Cores | 512GB DDR4-3200 ECC | PR2/C2 worker
len91 | Ubuntu 22.04.3 LTS | WD Black SN750 2TB WDS200T3X0C-00SJG0 | XFS | AMD Ryzen Threadripper PRO 3945WX 12-Cores | 256GB DDR4-3200 ECC | lotus-miner
len101 | Ubuntu 22.04.3 LTS | WD Black SN750 2TB WDS200T3X0C-00SJG0 | XFS | AMD Ryzen Threadripper PRO 3995WX 64-Cores | 1024GB DDR4-3200 ECC | RU/PC2 worker
In addition to `lotus` software (on all servers) and `boost` (lun91), `subspace` nodes are also installed on two workers, but separate drives are allocated for Yugabyte, and processor resources are not even half occupied. Sealing is carried out intermittently, from time to time.
>How does your boostd-data talks to yugabyte
I use boostd-data service:
```[Unit]
Description=boostd-data
After=network-online.target
After=yb-tserver.service
Requires=network-online.target
Wants=yb-tserver.service
[Service]
ExecStart=/usr/local/bin/boostd-data run yugabyte --hosts 192.168.11.91,192.168.11.14,192.168.11.101 --connect-string="postgresql://yugabyte:yugabyte@192.168.11.91:5433?sslmode=disable" --addr 0.0.0.0:8044
Restart=always
RestartSec=5
TimeoutStartSec=20
TimeoutStopSec=20
User=adm
Group=adm
Type=simple
[Install]
WantedBy=multi-user.target
```
I've tried using another option as well, `?sslmode=disable&load_balance=true`, wuth the same result.
>Boostd logs and boostd-data logs
[boostd-data.log](https://github.com/filecoin-project/boost/files/13675323/boostd-data.log)
boostd-data's log attached. The boost upgrade to v2.1.1 was carried out on 11.12.2023
Boostd log - please download from here: [https://disk.yandex.ru/d/ugDMZaikacDfVQ](https://disk.yandex.ru/d/ugDMZaikacDfVQ)
>Recent error, warning and INFO logs from Yugabyte
Attached.
[yb-master-tserver-logs.zip](https://github.com/filecoin-project/boost/files/13675625/yb-master-tserver-logs.zip)
### Logging Information
```text
Logs has been attached as files
```
### Repo Steps
1. Install lotus-miner v1.25.1+boost v2.2.1+YugabyteDB v2.18.2.1 (replication factor 3, 3 nodes)
2. Start all processes
3. Wait for the described symptoms to appear.
...
Contributor guide
Research direction
Start by correlating the attached boostd-data.log, boostd log, and yb-master-tserver-logs.zip with the provided boostd-data systemd unit and Yugabyte connection settings. Reproduce the failure using the listed Lotus, Boost, and Yugabyte versions, then determine why deal acceptance, retrievals, announcing, and the Piece Doctor page stop together. Done means the underlying failure is identified and those operations remain functional after the same workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100