elastic / elastic/beats

[Flaky Test] filebeat.tests.system.test_shutdown.Test test_shutdown_wait_ok_1_slab

Open
#51,528 1 comment 1 reaction 0 assignees View on GitHub
flaky-test needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

## Flaky Test

* **Test Name:** test_shutdown_wait_ok_1_slab
* **Scope:** filebeat.tests.system.test_shutdown.Test
* **File:** filebeat/tests/system/test_shutdown.py
* **Location:** filebeat/tests/system/test_shutdown.py:51
* **Buildkite Link:** https://buildkite.com/organizations/elastic/analytics/suites/beats/tests/d7b8b94b-2219-85ee-b9d4-544e362634da
* **Flaky Instances:** 1
* **Latest Occurrence:** 2026-06-25T04:33:03.790Z

### Details

```json
{
"id": "d7b8b94b-2219-85ee-b9d4-544e362634da",
"web_url": "https://buildkite.com/organizations/elastic/analytics/suites/beats/tests/d7b8b94b-2219-85ee-b9d4-544e362634da",
"scope": "filebeat.tests.system.test_shutdown.Test",
"name": "test_shutdown_wait_ok_1_slab",
"location": "filebeat/tests/system/test_shutdown.py:51",
"file_name": "filebeat/tests/system/test_shutdown.py",
"instances": 1,
"latest_occurrence_at": "2026-06-25T04:33:03.790Z",
"most_recent_instance_at": "2026-06-25T04:33:03.790Z",
"last_resolved_at": null,
"ownership_team_ids": [],
"failure_examples_count": 1
}
```

### Failure Examples

**Example 1:**
**Run:** https://api.buildkite.com/v2/analytics/organizations/elastic/suites/beats/runs/cc5703c9-217e-83ee-a80f-24f5f48bf27f
**Time:** 2026-06-25T04:06:48.823Z
**Stacktrace:**

```
a = (,)

@wraps(func)
def standalone_func(*a):
> return func(*(a + p.args), **p.kwargs)

../build/ve/linux/lib/python3.9/site-packages/parameterized/parameterized.py:518:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = , queue_type = 'slab'

@parameterized.expand(QUEUE_TYPES)
def test_shutdown_wait_ok(self, queue_type):
"""
Test stopping filebeat under load: wait for all events being published.
Runs under both queue implementations (memqueue and slabqueue).
"""

self.nasa_logs()

self.render_config_template(
path=os.path.abspath(self.working_dir) + "/log/*",
ignore_older="1h",
shutdown_timeout="10s",
rotate_every_kb=10000,
queue_type=queue_type,
)
filebeat = self.start_beat()

# Wait until first flush
self.wait_until(
lambda: self.log_contains_count("Publish event") > 200,
max_timeout=15)

filebeat.check_kill_and_wait()

self.get_log()
self.wait_log_contains(
"Output shutdown started.",
max_timeout=15)

self.wait_log_contains(
"Continue shutdown: All enqueued events have been published.",
max_timeout=15)

# validate registry entry offset matches last published event
registry = self.get_registry()
outputs = self.read_output()
offset = registry[0]["offset"]
assert len(registry) == 1

# we allow for a potential race in the harvester shutdown here.
# In some cases the registry offset might match the penultimate offset.

eol_offset = 1
if os.name == "nt":
eol_offset += 1

> assert (offset == (outputs[-1]["log.offset"] + eol_offset + len(outputs[-1]["message"])) or
offset == (outputs[-2]["log.offset"] + eol_offset + len(outputs[-2]["message"])))
E assert (219405 == ((219503 + 1) + 97) or 219405 == ((219405 + 1) + 97))
E + where 97 = len('204.212.132.27 - - [01/Jul/1995:00:34:04 -0400] "GET /images/KSC-logosmall.gif HTTP/1.0" 200 1204')
E + and 97 = len('204.212.132.27 - - [01/Jul/1995:00:34:04 -0400] "GET /images/NASA-logosmall.gif HTTP/1.0" 200 786')

tests/system/test_shutdown.py:99: AssertionError
```

Contributor guide

Open the contributing guide

Research direction

Start with filebeat/tests/system/test_shutdown.py at the test_shutdown_wait_ok method and line 51, then reproduce the slab queue failure using the linked Buildkite run. Trace the registry and output offsets around shutdown and verify the test passes reliably with the expected final or penultimate event offset.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.