apache / apache/doris

[Bug] [Config] Inconsistent defaults for storage flood stage thresholds between FE and BE

Open
#67,934 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

v3.1.4

### What's Wrong?

[Config] Inconsistent defaults for storage flood stage thresholds between FE and BE

Description

Apache Doris defines storage_flood_stage_usage_percent and storage_flood_stage_left_capacity_bytes in both FE and BE configuration. The two components use the same parameter names, but the default value of storage_flood_stage_usage_percent is inconsistent.

According to the official documentation:


Parameter | FE default | BE default
-- | -- | --
storage_flood_stage_usage_percent | 95 | 90
storage_flood_stage_left_capacity_bytes | 1 GiB | 1 GiB

References:

Problem

### Conflicting Configuration Defaults Between FE and BE

The same configuration parameters have **conflicting defaults in FE and BE**, which can mislead users when configuring a Doris cluster.

For example, at **92% disk usage**:
* **FE:** Stays normal (default threshold is **95%**).
* **BE:** Triggers flood-stage protection and returns errors (default threshold is **90%**).

#### Impact
This mismatch causes **unexpected query failures**. It makes disk protection behavior unpredictable and complicates both cluster configuration and troubleshooting.

### What You Expected?

The default values and descriptions of the corresponding FE and BE parameters should be consistent, or the documentation should clearly explain why they intentionally differ.

### How to Reproduce?

- set config::storage_flood_stage_usage_percent 92% as hard-coded.
- set storage_flood_stage_left_capacity_bytes to a high value enough for satisy this condition in data_dir.cpp:

```
bool DataDir::reach_capacity_limit(int64_t incoming_data_size) {
double used_pct = get_usage(incoming_data_size);
int64_t left_bytes = _available_bytes - incoming_data_size;
if (used_pct >= config::storage_flood_stage_usage_percent / 100.0 &&
left_bytes <= config::storage_flood_stage_left_capacity_bytes) {
LOG(WARNING) << "reach capacity limit. used pct: " << used_pct
<< ", left bytes: " << left_bytes << ", path: " << _path;
return true;
}
return false;
}
```

- then the backend will return an error. It is still treated as a query execution candidate, though, since exceedLimit continues to return false:
```
public boolean exceedLimit(boolean floodStage) {
if (LOG.isDebugEnabled()) {
LOG.debug("flood stage: {}, diskAvailableCapacityB: {}, totalCapacityB: {}",
floodStage, diskAvailableCapacityB, totalCapacityB);
}
if (floodStage) {
return diskAvailableCapacityB < Config.storage_flood_stage_left_capacity_bytes
&& this.getUsedPct() > (Config.storage_flood_stage_usage_percent / 100.0);
} else {
return diskAvailableCapacityB < Config.storage_min_left_capacity_bytes
|| this.getUsedPct() > (Config.storage_high_watermark_usage_percent / 100.0);
}
}
```

### Anything Else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Compare the defaults in FE Config.java and the BE implementation in data_dir.cpp, then read the linked FE and BE configuration documentation. Confirm whether the 95% and 90% values are intentional and trace the documented behavior around storage_flood_stage_usage_percent and storage_flood_stage_left_capacity_bytes. Done means the FE and BE defaults and descriptions are consistent, or their intentional difference is clearly documented, with tests or verification covering the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.