ChainSafe / ChainSafe/lodestar

[glamsterdam-devnet-8] Unknown sync gave up before gossip block came

Open
#10,018 1 comment 0 reactions 1 assignee Claimed by @twoeths View on GitHub
meta-bug
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 18h
Merged PRs (30d)
156

Description

### Describe the bug

log by root
```
2026-09-06 10:27:25.909 | Sep-06 03:27:25.907[chain] debug: Persisted payload envelope to db slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
-- | -- | --
  |   | 2026-09-06 10:27:25.909 | Sep-06 03:27:25.906[chain] debug: Persisted dataColumnSidecars to hot DB slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, dataColumnSidecars=128, custodyColumns=128, numBlobs=11 |  
  |   | 2026-09-06 10:27:25.891 | Sep-06 03:27:25.891[chain] verbose: Execution payload imported slot=170237, builderIndex=10044, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, blockHash=0x3ccdee7a83d51bbeca735129f9ddb2cc2d7093b0ebb898e863ce0ef78cdb84b5, delaySec=1.8880000114440918 |  
  |   | 2026-09-06 10:27:25.842 | Sep-06 03:27:25.842[network] debug: Received gossip payload envelope slot=170237, currentSlot=170237, peerId=16Uiu2HAmVHg5G6u7rvEuHZpBNUc7C3VxZuU38dtXC7NZrdt7oEsW, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, delaySec=1.8369998931884766 |  
  |   | 2026-09-06 10:27:25.353 | Sep-06 03:27:25.353[chain] debug: Cannot serve ExecutionPayloadEnvelopesByRoot: envelope not found slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, peer=16...L3XHzb, client=Lodestar |  
  |   | 2026-09-06 10:27:24.428 | Sep-06 03:27:24.428[chain] verbose: Completed getBlobsV2 for block slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.427 | Sep-06 03:27:24.427[chain] debug: getBlobsV2 result for block slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, result=success_resolved |  
  |   | 2026-09-06 10:27:24.411 | Sep-06 03:27:24.411[chain] debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=78, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=9, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.411 | Sep-06 03:27:24.411[network] debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=9, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAmLCFHKTFwgbPv4pHF7njWKqoDyrYDMt95axco35h1ckc1, delaySec=0.3970000743865967, secFromSlot=0.3989999294281006, gossipSubnet=78, columnIndex=78, recvToValLatency=0.0019998550415039062, recvToValidation=0.01399993896484375, validationTime=0.012000083923339844 |  
  |   | 2026-09-06 10:27:24.410 | Sep-06 03:27:24.410[chain] debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=36, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=8, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.410 | Sep-06 03:27:24.410[network] debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=8, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm9rBtzvh7qpG5ipH6vZ6v11ymwKJPfprNRVpqLiL3XHzb, delaySec=0.39800000190734863, secFromSlot=0.40000009536743164, gossipSubnet=36, columnIndex=36, recvToValLatency=0.002000093460083008, recvToValidation=0.012000083923339844, validationTime=0.009999990463256836 |  
  |   | 2026-09-06 10:27:24.407 | Sep-06 03:27:24.407[chain] debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=0, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=7, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.407 | Sep-06 03:27:24.407[network] debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=7, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAmLCFHKTFwgbPv4pHF7njWKqoDyrYDMt95axco35h1ckc1, delaySec=0.3970000743865967, secFromSlot=0.3989999294281006, gossipSubnet=0, columnIndex=0, recvToValLatency=0.0019998550415039062, recvToValidation=0.009999990463256836, validationTime=0.00800013542175293 |  
  |   | 2026-09-06 10:27:24.396 | Sep-06 03:27:24.396[chain] debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=13, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=6, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.396 | Sep-06 03:27:24.396[network] debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=6, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.2630000114440918, secFromSlot=0.375, gossipSubnet=13, columnIndex=13, recvToValLatency=0.1119999885559082, recvToValidation=0.13299989700317383, validationTime=0.020999908447265625 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[chain] debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=24, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=5, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[network] debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=5, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.2539999485015869, secFromSlot=0.375, gossipSubnet=24, columnIndex=24, recvToValLatency=0.12100005149841309, recvToValidation=0.1400001049041748, validationTime=0.01900005340576172 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[chain] debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=26, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=4, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[network] debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=4, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.32399988174438477, secFromSlot=0.375, gossipSubnet=26, columnIndex=26, recvToValLatency=0.051000118255615234, recvToValidation=0.07000017166137695, validationTime=0.01900005340576172 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[chain] debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=68, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=3, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[network] debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=3, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.31999993324279785, secFromSlot=0.375, gossipSubnet=68, columnIndex=68, recvToValLatency=0.05500006675720215, recvToValidation=0.07400012016296387, validationTime=0.01900005340576172 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[chain] debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=53, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=2, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[network] debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=2, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.312000036239624, secFromSlot=0.375, gossipSubnet=53, columnIndex=53, recvToValLatency=0.06299996376037598, recvToValidation=0.0820000171661377, validationTime=0.01900005340576172 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[chain] debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=28, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=1, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[network] debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=1, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.2560000419616699, secFromSlot=0.375, gossipSubnet=28, columnIndex=28, recvToValLatency=0.11899995803833008, recvToValidation=0.1380000114440918, validationTime=0.01900005340576172 |  
  |   | 2026-09-06 10:27:24.386 | Sep-06 03:27:24.380[chain] verbose: Trigger getBlobsV2 for block slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.386 | Error: Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[0-127] |  
  |   | 2026-09-06 10:27:24.386 | Sep-06 03:27:24.379[sync] debug: Ignoring unknown payload root after failed download slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.37599992752075195 - Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[0-127] |  
  |   | 2026-09-06 10:27:24.378 | Sep-06 03:27:24.376[sync] verbose: BlockInputSync.downloadPayload() slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.37599992752075195 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain] debug: Pruned block input slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain] debug: Persisted blockInput to db slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain] debug: Persist block to hot DB slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, inputType=no-data, numBlobs=11 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain] verbose: Block processed slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, delaySec=0.37400007247924805 |  
  |   | 2026-09-06 10:27:24.373 | Sep-06 03:27:24.373[chain] verbose: Strong block detected. Not override fcu call blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, slot=170237, reason=notProposerOfNextSlot |  
  |   | 2026-09-06 10:27:24.373 | Sep-06 03:27:24.373[chain] verbose: New chain head slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, delaySec=0.372999906539917 |  
  |   | 2026-09-06 10:27:24.372 | Sep-06 03:27:24.372[chain] verbose: Added block to forkchoice and state cache slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.278 | Sep-06 03:27:24.277[chain] verbose: SeenPayloadEnvelopeInput.add created new entry slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, daOutOfRange=false |  
  |   | 2026-09-06 10:27:24.278 | Error: Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[] |  
  |   | 2026-09-06 10:27:24.278 | Sep-06 03:27:24.276[sync] debug: Ignoring unknown payload root after failed download slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.2669999599456787 - Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[] |  
  |   | 2026-09-06 10:27:24.272 | Sep-06 03:27:24.267[sync] verbose: BlockInputSync.downloadPayload() slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.2669999599456787 |  
  |   | 2026-09-06 10:27:24.265 | Sep-06 03:27:24.265[sync] verbose: Added new payload rootHex to BlockInputSync.pendingPayloads slot=170237, delaySec=0.2650001049041748, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, peerIdStr=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ |  
  |   | 2026-09-06 10:27:24.258 | Sep-06 03:27:24.258[sync] debug: ignored Banning unknown block slot=unknown, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, peerIdStrings=16...h1ckc1 |  
  |   | 2026-09-06 10:27:24.258 | Sep-06 03:27:24.258[sync] debug: Removing bad/unknown/incomplete BlockInputSyncCacheItem slot=unknown, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.258 | Error: Error fetching UnknownBlockRoot slot=undefined root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[0-127] |  
  |   | 2026-09-06 10:27:24.258 | Sep-06 03:27:24.258[sync] debug: Ignoring unknown block root after many failed downloads slot=unknown, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingBlocks=1 - Error fetching UnknownBlockRoot slot=undefined root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[0-127] |  
  |   | 2026-09-06 10:27:24.255 | Sep-06 03:27:24.255[sync] verbose: BlockInputSync.downloadBlock() slot=unknown, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingBlocks=1 |  
  |   | 2026-09-06 10:27:24.255 | Sep-06 03:27:24.255[sync] verbose: Added new rootHex to BlockInputSync.pendingBlocks root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, peerIdStr=16Uiu2HAmLCFHKTFwgbPv4pHF7njWKqoDyrYDMt95axco35h1ckc1
```

log by slot

```
2026-09-06 10:27:24.386 | Sep-06 03:27:24.379[sync] debug: Ignoring unknown payload root after failed download slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.37599992752075195 - Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[0-127] |  
-- | -- | --
  |   | 2026-09-06 10:27:24.378 | Sep-06 03:27:24.376[sync] verbose: BlockInputSync.downloadPayload() slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.37599992752075195 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain] debug: Pruned block input slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain] debug: Persisted blockInput to db slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain] debug: Persist block to hot DB slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, inputType=no-data, numBlobs=11 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain] verbose: Block processed slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, delaySec=0.37400007247924805 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain] debug: Imported block slot=170237, recvToValLatency=0.11500000953674316, recvToValidation=0.11700010299682617, validationTime=0.002000093460083008 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain] verbose: After importBlock caching postState without SSZ cache slot=170237 |  
  |   | 2026-09-06 10:27:24.373 | Sep-06 03:27:24.373[chain] verbose: Strong block detected. Not override fcu call blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, slot=170237, reason=notProposerOfNextSlot |  
  |   | 2026-09-06 10:27:24.373 | Sep-06 03:27:24.373[chain] verbose: New chain head slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, delaySec=0.372999906539917 |  
  |   | 2026-09-06 10:27:24.372 | Sep-06 03:27:24.372[chain] verbose: Added block to forkchoice and state cache slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.372 | Sep-06 03:27:24.372[chain] debug: Verified block signatures slot=170237, recvToValLatency=0.10500001907348633, recvToValidation=0.11500000953674316, validationTime=0.009999990463256836 |  
  |   | 2026-09-06 10:27:24.362 | Sep-06 03:27:24.362[chain] debug: Verified block state transition slot=170237, recvToValLatency=0.023000001907348633, recvToValidation=0.10500001907348633, validationTime=0.0820000171661377 |  
  |   | 2026-09-06 10:27:24.286 | Sep-06 03:27:24.279[network] debug: Validated gossip block slot=170237, blockRoot=0xab60…92c0, timeCreatedSec=1788665244.257, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.2569999694824219, recvToValLatency=0.019999980926513672, recvToValidation=0.02200007438659668, validationTime=0.002000093460083008, skippedSlots=0 |  
  |   | 2026-09-06 10:27:24.278 | Sep-06 03:27:24.277[chain] verbose: SeenPayloadEnvelopeInput.add created new entry slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, daOutOfRange=false |  
  |   | 2026-09-06 10:27:24.278 | Sep-06 03:27:24.277[chain] debug: Attempt to cache block but is already cached on BlockInput slot=170237, blockRoot=0xab60…92c0, timeCreatedSec=1788665244.257 |  
  |   | 2026-09-06 10:27:24.278 | Sep-06 03:27:24.277[network] debug: Received gossip block slot=170237, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.2569999694824219, recvToValLatency=0.019999980926513672 |  
  |   | 2026-09-06 10:27:24.278 | Error: Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[] |  
  |   | 2026-09-06 10:27:24.278 | Sep-06 03:27:24.276[sync] debug: Ignoring unknown payload root after failed download slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.2669999599456787 - Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[] |  
  |   | 2026-09-06 10:27:24.272 | Sep-06 03:27:24.267[sync] verbose: BlockInputSync.downloadPayload() slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.2669999599456787 |  
  |   | 2026-09-06 10:27:24.265 | Sep-06 03:27:24.265[sync] verbose: Added new payload rootHex to BlockInputSync.pendingPayloads slot=170237, delaySec=0.2650001049041748, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, peerIdStr=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ |  
  |   | 2026-09-06 10:27:24.001 | Sep-06 03:27:24.000[chain] verbose: Clock slot slot=170237
```

### Expected behavior

the pending block was removed 0.27s into the slot and recreated multiple times
we should keep the block (up until perhaps 1 slot = 12 seconds) and accumulate peer ids so that it's easier to download later
instead of giving up too early with only 1 peer tried

### Steps to reproduce

_No response_

### Additional context

_No response_

### Operating system

Linux

### Lodestar version or commit hash

unstable

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.