erigontech / erigontech/erigon

Bad UX when activating blobs archive (bug?)

Open
#13,737 0 comments 0 reactions 0 assignees View on GitHub
UX
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
465

Description

Today's main, I just enabled blobs archive in an existing mainnet node with `--caplin.blobs-archive`.

At first it seemed it was going to download +300GB according to the logs:

```
[INFO] [02-07|20:18:29.226] [Gossip] Subscribed to topic topic=/eth2/6a95a1a9/beacon_block/ssz_snappy
[INFO] [02-07|20:18:29.226] [Gossip] Subscribed to topic topic=/eth2/6a95a1a9/blob_sidecar_8/ssz_snappy
[INFO] [02-07|20:18:40.191] [1/6 OtterSync] Downloading progress="(1421/1421 files) 86.75% - 1.7TB/2.0TB" time-left=999hrs:99m total-time=40s download-rate=0B/s completion-rate=0B/s alloc=12.4GB sys
=12.9GB
[INFO] [02-07|20:18:40.235] [snapshots] no progress yet files=186 list=v1-009000-009100-beaconblocks.seg,v1-009860-009870-blobsidecars.seg,v1-004400-004500-beaconblocks.seg,v1-009850-009860-beaconbl
ocks.seg,v1-010710-010720-beaconblocks.seg,...
[INFO] [02-07|20:19:00.191] [1/6 OtterSync] Downloading progress="(1421/1421 files) 86.84% - 1.7TB/2.0TB" time-left=0hrs:47m total-time=1m0s download-rate=94.3MB/s completion-rate=94.3MB/s alloc=12.
8GB sys=13.2GB
[INFO] [02-07|20:19:00.278] [snapshots] no progress yet files=173 list=v1-010470-010480-blobsidecars.seg,v1-010690-010700-beaconblocks.seg,v1-010400-010410-beaconblocks.seg,v1-008890-008900-blobside
cars.seg,v1-010670-010680-blobsidecars.seg,...
[INFO] [02-07|20:19:20.191] [1/6 OtterSync] Downloading progress="(1421/1421 files) 86.94% - 1.7TB/2.0TB" time-left=0hrs:41m total-time=1m20s download-rate=108.3MB/s completion-rate=107.9MB/s alloc=
13.3GB sys=13.7GB
```

and that's fine, that's something expected given the flag was explicitly added. However after finishing that, it added more +300GB of downloads, see the next logs:

```
[INFO] [02-07|21:08:20.256] [1/6 OtterSync] Downloading progress="(1421/1637 files) 100.00% - 2.0TB/2.0TB" time-left=0hrs:0m total-time=50m20s download-rate=51.2KB/s completion-rate=51.2KB/s alloc=1
2.5GB sys=18.9GB
[INFO] [02-07|21:08:20.278] [snapshots] no metadata yet files=216 list=v1-005900-006000-beaconblocks.seg,v1-006300-006400-beaconblocks.seg,v1-004100-004200-beaconblocks.seg,v1-010140-010150-blobside
cars.seg,v1-010430-010440-blobsidecars.seg,...
[INFO] [02-07|21:08:28.218] P2P app=caplin peers=4
[INFO] [02-07|21:08:40.192] [1/6 OtterSync] Downloading progress="(1421/1637 files) 100.00% - 2.0TB/2.0TB" time-left=0hrs:0m total-time=50m40s download-rate=12.8KB/s completion-rate=12.8KB/s alloc=1
5.1GB sys=18.9GB
[INFO] [02-07|21:08:40.234] [snapshots] no progress yet files=181 list=v1-009110-009120-blobsidecars.seg,v1-009080-009090-blobsidecars.seg,v1-010310-010320-beaconblocks.seg,v1-009970-009980-blobside
cars.seg,v1-010300-010310-beaconblocks.seg,...
[INFO] [02-07|21:09:00.191] [1/6 OtterSync] Downloading progress="(1637/1637 files) 85.10% - 2.0TB/2.3TB" time-left=19hrs:9m total-time=51m0s download-rate=5.2MB/s completion-rate=5.0MB/s alloc=15.5
GB sys=18.9GB
[INFO] [02-07|21:09:00.192] [p2p] GoodPeers eth68=3
[INFO] [02-07|21:09:00.255] [txpool] stat pending=0 baseFee=0 queued=0 alloc=15.5GB sys=18.9GB
[INFO] [02-07|21:09:00.268] [snapshots] no progress yet files=8 list=v1-010420-010430-blobsidecars.seg,v1-010730-010740-blobsidecars.seg,v1-010720-010730-beaconblocks.seg,v1-010460-010470-blobsideca
rs.seg,v1-009550-009560-blobsidecars.seg,...
[INFO] [02-07|21:09:01.017] [mem] memory stats Rss=21.7GB Size=0B Pss=21.7GB SharedClean=2.7MB SharedDirty=0B PrivateClean=7.8GB PrivateDirty=13.8GB Referenced=11.1GB Anonymous=12.9GB Swap=
4.8GB alloc=15.5GB sys=18.9GB
[INFO] [02-07|21:09:20.207] [1/6 OtterSync] Downloading progress="(1637/1637 files) 85.19% - 2.0TB/2.3TB" time-left=0hrs:57m total-time=51m20s download-rate=103.3MB/s completion-rate=103.3MB/s alloc
=15.8GB sys=18.9GB
[INFO] [02-07|21:09:21.146] [snapshots] no progress yet files=5 list=v1-010460-010470-blobsidecars.seg,v1-010400-010410-blobsidecars.seg,v1-010680-010690-blobsidecars.seg,v1-010420-010430-blobsideca
rs.seg,v1-010730-010740-blobsidecars.seg
[INFO] [02-07|21:09:28.218] P2P app=caplin peers=0
[INFO] [02-07|21:09:40.191] [1/6 OtterSync] Downloading progress="(1637/1637 files) 85.25% - 2.0TB/2.3TB" time-left=1hrs:13m total-time=51m40s download-rate=80.0MB/s completion-rate=80.1MB/s alloc=1
6.1GB sys=18.9GB
```

Not sure if there is a bug on accounting of initial added download or it is expected that those files are added in batches, in this case it seems at least UX could be improved because we are surprising the user: "oh right, it'll download +300GB... good, it finished, oh wait it is going to download +300GB, how much is the total?"

Contributor guide

Open the contributing guide

Research direction

Start by reproducing activation with --caplin.blobs-archive and inspect the OtterSync and snapshots log output shown in the report. Compare the initial and subsequently added file batches to determine whether progress accounting is incorrect or downloads are intentionally staged; done means the user can see an accurate overall download expectation without surprising extra batches.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, blockchain
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.