redpanda-data / redpanda-data/redpanda

Enforce maximum allocation size in ducktape

Open
#8,320 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/tests kind/enhance
Dominant language
C++
Stars
12.6k
Forks
792
PR merge metrics
No merged PRs in 30d

Description

After startup, redpanda must not allocate large contiguous memory areas (a suggested maximum is 128 KiB), as these are likely to fail due to fragmentation, even if sufficient memory is available to otherwise satisfy the allocation.

To enforce this, we can turn on the seastar large allocation warning (see seastar::memory::set_large_allocation_warning_threshold), which will emit a warning and backtrace into the log, and then fail tests using bad log lines (should happen automatically as this is a new error/warning output).

This has a risk of introducing a new source of non-determinism in tests, since allocation sizes may in some cases be non-deterministic (e.g., because some queue sizes may depend on timing, or exactly when task suspension occurs which itself depends on timing, and some sizes will depend on random factors like partition allocation).

One way to try to reduce the non-determinism would be to run with a threshold of 2X for the every-PR tests, and a lower threshold of X for some nightly tests, with the idea that this isolates the on-PR tests from most of the flakiness where it is the most annoying, since tests that are near the failure threshold will fail first in nightly tests.

At these are "easy to debug & fix" failures since the backtrace points to exactly the offending allocation and the fix is usually straightforward enough (or, perhaps, in some cases we can exclude certain allocations from the diagnostic if they are harder to fix).

JIRA Link: CORE-1144

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Redpanda test configuration and the startup path where seastar::memory::set_large_allocation_warning_threshold can be enabled. Review how bad log lines are handled in every-PR and nightly tests, then verify that allocations above the selected threshold produce actionable backtraces without introducing unacceptable test flakiness.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.