nervosnetwork / nervosnetwork/ckb
RocksDB L5+L6 compaction may require more than ~192 GiB free disk space due to high read/write amplification
- Dominant language
- Rust
- Stars
- 1.2k
- Forks
- 266
- Avg merge
- 10d 5h
- Merged PRs (30d)
- 4
Description
## Bug Report
### Current Behavior
A `ckb v0.205.0` testnet node repeatedly hit RocksDB compaction errors during a large `L5 + L6` compaction.
Relevant logs:
```text
2026/04/03-12:11:49.105337 51 [ERROR] [file/sst_file_manager_impl.cc:191] free space [196756488192 bytes] is less than needed headroom [206219919386 bytes]
2026/04/03-12:11:49.105408 51 [ERROR] [db/db_impl/db_impl_compaction_flush.cc:3159] Waiting after background compaction error: Compaction too large: , Accumulated
background error counts: 50193
[JOB 15] Compacting 1@5 + 3144@6 files to L6, score 0.41
2026/04/03-14:02:40.814464 51 [ERROR] [file/sst_file_manager_impl.cc:191] free space [196766220288 bytes] is less than needed headroom [206244698514 bytes]
2026/04/03-14:02:40.814552 51 [db/db_impl/db_impl_compaction_flush.cc:51] Cancelled compaction because not enough room
2026/04/03-14:02:40.814558 51 [ERROR] [db/db_impl/db_impl_compaction_flush.cc:3159] Waiting after background compaction error: Compaction too large: , Accumulated
background error counts: 508
```
This appears to be a compaction for column family 2 (COLUMN_BLOCK_BODY). During this period, the node kept generating new .sst files and disk usage grew very quickly.
From the logs, RocksDB needed about 206244698514 bytes of free space, which is about 192 GiB, while available free space was about 196766220288 bytes, about 183 GiB.
### Expected Behavior
### Environment
- **CKB version**: v0.205.0
- **Chain**: testnet
- **Operating system**: ubuntu
- **Arch**: x86_64
- **Installation**: [GitHub Release, Built from source]
Contributor guide
Assessment
This issue has not been assessed yet.