elastic / elastic/beats

[Flaky Test] x-pack.filebeat.tests.system.test_xpack_modules.XPackTest test_fileset_file_205_google_workspace

Open
#49,958 1 comment 0 reactions 0 assignees View on GitHub
flaky-test Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

## Flaky Test

* **Test Name:** test_fileset_file_205_google_workspace
* **Scope:** x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
* **File:** filebeat/tests/system/test_modules.py
* **Location:** filebeat/tests/system/test_modules.py:137
* **Buildkite Link:** https://buildkite.com/organizations/elastic/analytics/suites/beats/tests/0485bf8e-b4ec-82c8-891e-8cddd44c7b09
* **Flaky Instances:** 1
* **Latest Occurrence:** 2026-04-07T10:26:04.625Z

### Details

```json
{
"id": "0485bf8e-b4ec-82c8-891e-8cddd44c7b09",
"web_url": "https://buildkite.com/organizations/elastic/analytics/suites/beats/tests/0485bf8e-b4ec-82c8-891e-8cddd44c7b09",
"scope": "x-pack.filebeat.tests.system.test_xpack_modules.XPackTest",
"name": "test_fileset_file_205_google_workspace",
"location": "filebeat/tests/system/test_modules.py:137",
"file_name": "filebeat/tests/system/test_modules.py",
"instances": 1,
"latest_occurrence_at": "2026-04-07T10:26:04.625Z",
"most_recent_instance_at": "2026-04-07T10:26:04.625Z",
"last_resolved_at": null,
"ownership_team_ids": [],
"failure_examples_count": 1
}
```

### Failure Examples

**Example 1:**
**Run:** https://api.buildkite.com/v2/analytics/organizations/elastic/suites/beats/runs/97605a8c-8ecf-8386-b7fb-4963b2a11b51
**Time:** 2026-04-07T07:42:15.589Z
**Stacktrace:**

```
a = (,)

@wraps(func)
def standalone_func(*a):
> return func(*(a + p.args), **p.kwargs)

../../build/ve/linux/lib/python3.9/site-packages/parameterized/parameterized.py:518:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../filebeat/tests/system/test_modules.py:159: in test_fileset_file
self.run_on_file(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = , module = 'google_workspace', fileset = 'admin'
test_file = '/opt/buildkite-agent/builds/bk-agent-prod-gcp-1775547366362433884/elastic/beats-xpack-filebeat/x-pack/filebeat/module/google_workspace/admin/test/admin-application-test.json.log'
cfgfile = '/opt/buildkite-agent/builds/bk-agent-prod-gcp-1775547366362433884/elastic/beats-xpack-filebeat/x-pack/filebeat/build/system-tests/run/test_xpack_modules.XPackTest.test_fileset_file_205_google_workspace914/filebeat.yml'

def run_on_file(self, module, fileset, test_file, cfgfile):
print("Testing {}/{} on {}".format(module, fileset, test_file))

self.assert_explicit_ecs_version_set(module, fileset)

try:
resp = self.es.indices.delete_data_stream(name=self.index_name)
except NotFoundError:
pass

self.wait_until(lambda: not self.es.indices.exists(index=self.index_name))

cmd = [
self.filebeat, "--systemTest",
"-d", "*", "--once",
"-c", cfgfile,
"-E", "setup.ilm.enabled=false",
"--modules={}".format(module),
"-M", "{module}.*.enabled=false".format(module=module),
"-M", "{module}.{fileset}.enabled=true".format(
module=module, fileset=fileset),
"-M", "{module}.{fileset}.var.input=file".format(
module=module, fileset=fileset),
"-M", "*.*.input.close_eof=true",
]

# if the test file contains '.journal', later it will try to remove
# the '--once' flag and the journald input will be used,
# so there is nothing to do here.
if log_as_filestream() and ".journal" not in test_file:
cmd.append("-E")
cmd.append("features.log_input_run_as_filestream.enabled=true")
cmd.append("-M")
cmd.append("{module}.{fileset}.input.id='id{module}-{fileset}'".format(module=module, fileset=fileset))
cmd.remove("--once")
# allow connecting older versions of Elasticsearch
if os.getenv("TESTING_FILEBEAT_ALLOW_OLDER"):
cmd.extend(["-E", "output.elasticsearch.allow_older_versions=true"])

# Based on the convention that if a name contains -json the json format is needed. Currently used for LS.
if "-json" in test_file:
cmd.append("-M")
cmd.append("{module}.{fileset}.var.format=json".format(
module=module, fileset=fileset))

if ".journal" in test_file:
cmd.remove("--once")
cmd.append("-M")
cmd.append("{module}.{fileset}.var.use_journald=true".format(
module=module, fileset=fileset))
cmd.append("-M")
cmd.append("{module}.{fileset}.input.paths=[{test_file}]".format(
module=module, fileset=fileset, test_file=test_file))
else:
cmd.append("-M")
cmd.append("{module}.{fileset}.var.paths=[{test_file}]".format(
module=module, fileset=fileset, test_file=test_file))

output_path = os.path.join(self.working_dir)
# Runs inside a with block to ensure file is closed afterwards
with open(os.path.join(output_path, "output.log"), "ab") as output:
output.write(bytes("Command run: ", "utf-8"))
output.write(bytes(" ".join(cmd) + "\n\n", "utf-8"))
output.flush()

# Use a fixed timezone so results don't vary depending on the environment
# Don't use UTC to avoid hiding that non-UTC timezones are not being converted as needed,
# this can happen because UTC uses to be the default timezone in date parsers when no other
# timezone is specified.
local_env = os.environ.copy()
local_env["TZ"] = 'Etc/GMT+2'

proc = subprocess.Popen(cmd,
env=local_env,
stdin=None,
stdout=output,
stderr=subprocess.STDOUT,
bufsize=0)
# The journald input (used by some modules like 'system') does not
# support the --once flag, hence we run Filebeat for at most
# 15 seconds, if it does not finish, then kill the process.
# If for any reason the Filebeat process gets stuck, only SIGKILL
# will terminate it. We use SIGKILL to avoid leaking any running
# process that could interfere with other tests
try:
proc.wait(15)
except subprocess.TimeoutExpired:
# Send SIGKILL
proc.kill()

# List of errors to check in filebeat output logs
errors = ["error loading pipeline for fileset"]
# Checks if the output of filebeat includes errors
contains_error, error_line = file_contains(
os.path.join(output_path, "output.log"), errors)
assert contains_error is False, "Error found in log:{}".format(
error_line)

# Make sure index exists
self.wait_until(lambda: self.es.indices.exists(index=self.index_name),
name="indices present for {}".format(test_file))

self.es.indices.refresh(index=self.index_name)
# Loads the first 100 events to be checked
res = self.es.search(index=self.index_name, query={"match_all": {}},
size=100, sort={"log.offset": {"order": "asc"}})
objects = [o["_source"] for o in res["hits"]["hits"]]
> assert len(objects) > 0
E assert 0 > 0
E + where 0 = len([])

../../filebeat/tests/system/test_modules.py:272: AssertionError
```

Contributor guide

Open the contributing guide

Research direction

Start with filebeat/tests/system/test_modules.py around lines 137 and 159, then run XPackTest.test_fileset_file_205_google_workspace and review the linked Buildkite failure. Investigate why the test produces zero indexed objects; done means the test reliably observes events for the Google Workspace admin fileset.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.