redpanda-data / redpanda-data/redpanda
Enforce maximum allocation size in ducktape
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 12.6k
- Forks
- 792
- PR merge metrics
- No merged PRs in 30d
Description
After startup, redpanda must not allocate large contiguous memory areas (a suggested maximum is 128 KiB), as these are likely to fail due to fragmentation, even if sufficient memory is available to otherwise satisfy the allocation.
To enforce this, we can turn on the seastar large allocation warning (see seastar::memory::set_large_allocation_warning_threshold), which will emit a warning and backtrace into the log, and then fail tests using bad log lines (should happen automatically as this is a new error/warning output).
This has a risk of introducing a new source of non-determinism in tests, since allocation sizes may in some cases be non-deterministic (e.g., because some queue sizes may depend on timing, or exactly when task suspension occurs which itself depends on timing, and some sizes will depend on random factors like partition allocation).
One way to try to reduce the non-determinism would be to run with a threshold of 2X for the every-PR tests, and a lower threshold of X for some nightly tests, with the idea that this isolates the on-PR tests from most of the flakiness where it is the most annoying, since tests that are near the failure threshold will fail first in nightly tests.
At these are "easy to debug & fix" failures since the backtrace points to exactly the offending allocation and the fix is usually straightforward enough (or, perhaps, in some cases we can exclude certain allocations from the diagnostic if they are harder to fix).
JIRA Link: CORE-1144
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Redpanda test configuration and the startup path where seastar::memory::set_large_allocation_warning_threshold can be enabled. Review how bad log lines are handled in every-PR and nightly tests, then verify that allocations above the selected threshold produce actionable backtraces without introducing unacceptable test flakiness.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100