Blockstream / Blockstream/esplora
Electrs gets stuck during Bitcoin mainnet indexing on high-spec hardware
- Dominant language
- JavaScript
- Stars
- 1.3k
- Forks
- 514
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 16
Description
## Description
Liquid is working fine and Bitcoind syncs successfully to latest block but for the Bitcoin case, Electrs repeatedly gets stuck in uninterruptible sleep state during Bitcoin mainnet indexing, despite using high-performance hardware. The process appears to be waiting indefinitely on disk I/O operations during database compaction cycles.
## Environment
### Hardware:
Instance: AWS EC2 r6i.2xlarge
- 8 vCPUs
- 64GB RAM
Storage: EBS gp3 volume
- Size: 3TB
- IOPS: 16,000
- Throughput: 1,000 MB/s
## Configuration
```
services:
esplora-bitcoin:
image: blockstream/esplora:latest
command: ["/srv/explorer/run.sh", "bitcoin-mainnet", "explorer"]
environment:
- ENABLE_LIGHTMODE=1
ports:
- "50001:50001"
- "8080:80"
volumes:
- ./data_bitcoin_mainnet:/data
restart: unless-stopped
stdin_open: true
tty: true
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "3"
networks:
- bitcoin-net
esplora-liquid:
image: blockstream/esplora:latest
command: ["/srv/explorer/run.sh", "liquid-mainnet", "explorer"]
ports:
- "50002:50001"
- "8082:80"
volumes:
- ./data_liquid_mainnet:/data
restart: unless-stopped
stdin_open: true
tty: true
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "3"
networks:
- liquid-net
networks:
bitcoin-net:
liquid-net:
```
## Issue Details
### What works:
- ✅ Liquid explorer: Indexes and runs successfully
- ✅ Bitcoin daemon (bitcoind): Syncs to 100% without issues
- ✅ Disk performance: Confirmed with iostat showing 95-96% utilization during operations
### What fails:
- ❌ Bitcoin electrs indexing: Gets stuck repeatedly during indexing process
- ❌ Process state: Enters "D" state (uninterruptible sleep) and remains stuck for days
- ❌ Each container restart causes electrs to start indexing from block 0 again
## Observed Behavior
1. Bitcoind syncs successfully to latest block (e.g., block 919,039)
2. Electrs starts indexing: `INFO hash=... height=919038 (919039 left to index)`
3. Electrs process shows high CPU usage initially (20-25%)
4. After some time, process enters "D" state with 0% CPU:
```
PID USER STATE %CPU %MEM COMMAND
76 root D 0.0 22.1 electrs
```
5. Process remains stuck in D state for 24-48+ hours
6. No progress in logs - last log entry remains unchanged for days
7. Database grows to ~1.1TB but indexing never completes
## Process State Details
When stuck, `top` output shows:
```
76 root 20 0 14.6g 13.7g 4352 D 0.0 22.1 10:16 electrs
```
- State: D (Uninterruptible Sleep - waiting on disk I/O)
- CPU: 0.0% (completely blocked)
- Memory: Normal usage
## Disk I/O During Stuck State
`iostat -x` output during stuck periods:
```
Device r/s rkB/s w/s wkB/s %util
nvme0n1 1687 11754 10.5 54 95-96%
```
1. Any insights into if what we are seeing is normal?
2. What is the recommended specs?
3. Is it possible to get an pre indexed version of the db and start for there?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.