apache / apache/doris

[Bug] All spill files land on one disk when several spill disks are configured

Open
#67,062 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

master

### What's Wrong?

On a backend configured with several spill disks, all spill files created within any two-second window land on the same disk while the other spill disks stay idle. Spill throughput is therefore capped at one disk's bandwidth, and that disk reaches its capacity limit long before the others.

The disk is chosen deterministically. `_get_stores_for_spill()` sorts candidates by usage ascending:

```cpp
// be/src/exec/spill/spill_file_manager.cpp
std::ranges::sort(stores_with_usage, [](auto&& a, auto&& b) { return a.second < b.second; });
```

and `create_spill_file()` always takes the front entry:

```cpp
// Select the first available data dir (sorted by usage ascending)
SpillDataDir* data_dir = data_dirs.front();
```

Sorting by usage is the right intent, but the usage value it sorts on is stale. `SpillDataDir::_get_disk_usage()` is derived from `_available_bytes`:

```cpp
double _get_disk_usage(int64_t incoming_data_size) const {
return _disk_capacity_bytes == 0
? 0
: (double)(_disk_capacity_bytes - _available_bytes + incoming_data_size) /
(double)_disk_capacity_bytes;
}
```

and `_available_bytes` is only refreshed by `SpillDataDir::update_capacity()`, which runs on the GC thread once per `config::spill_gc_interval_ms` (2s by default):

```cpp
void SpillFileManager::_spill_gc_thread_callback() {
while (!_stop_background_threads_latch.wait_for(
std::chrono::milliseconds(config::spill_gc_interval_ms))) {
gc(config::spill_gc_work_time_ms);
for (auto& [path, dir] : _spill_store_map) {
static_cast(dir->update_capacity());
}
}
}
```

Note that `_spill_data_bytes`, which *is* updated on every write, does not feed into `_get_disk_usage()`. So nothing the manager hands out during those two seconds changes the ordering: every caller computes the same usage vector, sorts it the same way, and picks the same winner.

This is easy to hit in practice — a single spilling query creates one spill file per partition per pipeline task, so hundreds of `create_spill_file()` calls can fall inside one refresh window.

### What You Expected?

Spill files should be spread across the spill disks that are effectively equally empty, so that all configured spill disks contribute bandwidth and fill at a comparable rate. A disk that is genuinely emptier than the rest should still be preferred.

### How to Reproduce?

Configure a backend with several spill disks of similar free space, then run a query that spills heavily:

```sql
set enable_spill = true;
set enable_force_spill = true;

select l_orderkey, count(*) from lineitem group by l_orderkey;
```

Watch the per-disk `spill_disk_data_size` gauges (or the on-disk `spill/` directories). The bytes concentrate on one disk instead of being spread across them.

### Anything Else?

StarRocks solves the same problem by picking a random start index and walking the directory list from there (`be/src/compute_env/spill/dir_manager.cpp`). Randomizing only among the disks that a stale snapshot cannot tell apart keeps Doris's existing "prefer the emptiest disk" behaviour intact.

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start in be/src/exec/spill/spill_file_manager.cpp by tracing _get_stores_for_spill(), create_spill_file(), and the GC callback that refreshes SpillDataDir capacity. Reproduce with enable_spill and enable_force_spill using the lineitem aggregation, then inspect spill_disk_data_size or spill/ directories. Done means comparable spill distribution across equally empty disks while still preferring a genuinely emptier disk.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.