elastic / elastic/integrations

[bug-hunter] Buildkite affected-file filter misclassifies run_ci_python_scripts_tests.sh changes

Open
#19,268 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

## Impact
Changes to `.buildkite/scripts/run_ci_python_scripts_tests.sh` are treated as package-affecting changes, which unnecessarily triggers full package test fan-out instead of only CI-script checks. This increases CI time/cost and can block contributors with unrelated package test failures.

## Reproduction Steps
1. Run this standalone repro script (written against current `main`) to evaluate the same allowlist regex used by `is_pr_affected` in `.buildkite/scripts/common.sh`:

```bash
chmod +x /tmp/gh-aw/agent/repro_ci_non_package_bug.sh
/tmp/gh-aw/agent/repro_ci_non_package_bug.sh
```

2. Observe output.

## Expected vs Actual
**Expected:** `.buildkite/scripts/run_ci_python_scripts_tests.sh` should be treated like other non-package CI-only scripts in the allowlist path, so it does not mark packages as affected.

**Actual:** The script is not allowlisted; the same `grep -E -v "\$\{non_package_regex}"` logic used in `is_pr_affected` treats it as a non-allowlisted change and marks the PR as package-affecting.

Actual output from reproduction:

```text
non_package_regex=^(packages/|\.buildkite/pipeline\.backport\.yml|\.buildkite/pipeline\.publish\.yml|\.buildkite/pipeline\.serverless\.yml|\.buildkite/pipeline\.schedule-daily\.yml|\.buildkite/pipeline\.schedule-weekly\.yml|\.buildkite/pull-requests\.json|\.buildkite/scripts/backport_branch\.sh|\.buildkite/scripts/build_packages\.sh|\.buildkite/scripts/packages/.+\.sh|\.github/dependabot\.yml|\.github/workflows/|\.github/stale\.yml|\.github/CODEOWNERS|\.github/ISSUE_TEMPLATE/|\.github/PULL_REQUEST_TEMPLATE\.md|\.mergify\.yml|catalog-info\.yaml|docs/|CODE_OF_CONDUCT\.md|CONTRIBUTING\.md|README\.md|\.agents/skills/|dev/scripts/|\.buildkite/scripts/run_dev_scripts_tests\.sh)
PASS: dev script is allowlisted
BUG: ci python script is NOT allowlisted
BUG TRIGGERED: grep -v treats ci python script as non-package change
```

## Failing Test
```bash
#!/usr/bin/env bash
set -euo pipefail

# Reproduce non-package allowlist behavior from .buildkite/scripts/common.sh (is_pr_affected).
non_package_patterns=(
'packages/'
'\.buildkite/pipeline\.backport\.yml'
'\.buildkite/pipeline\.publish\.yml'
'\.buildkite/pipeline\.serverless\.yml'
'\.buildkite/pipeline\.schedule-daily\.yml'
'\.buildkite/pipeline\.schedule-weekly\.yml'
'\.buildkite/pull-requests\.json'
'\.buildkite/scripts/backport_branch\.sh'
'\.buildkite/scripts/build_packages\.sh'
'\.buildkite/scripts/packages/.+\.sh'
'\.github/dependabot\.yml'
'\.github/workflows/'
'\.github/stale\.yml'
'\.github/CODEOWNERS'
'\.github/ISSUE_TEMPLATE/'
'\.github/PULL_REQUEST_TEMPLATE\.md'
'\.mergify\.yml'
'catalog-info\.yaml'
'docs/'
'CODE_OF_CONDUCT\.md'
'CONTRIBUTING\.md'
'README\.md'
'\.agents/skills/'
'dev/scripts/'
'\.buildkite/scripts/run_dev_scripts_tests\.sh'
)

non_package_regex="^($(IFS='|'; echo "\$\{non_package_patterns[*]}"))"
echo "non_package_regex=$non_package_regex"

dev_script=".buildkite/scripts/run_dev_scripts_tests.sh"
ci_script=".buildkite/scripts/run_ci_python_scripts_tests.sh"

if printf '%s\n' "$dev_script" | grep -E "$non_package_regex" >/dev/null; then
echo "PASS: dev script is allowlisted"
else
echo "FAIL: dev script should be allowlisted"
exit 1
fi

if printf '%s\n' "$ci_script" | grep -E "$non_package_regex" >/dev/null; then
echo "UNEXPECTED: ci python script is allowlisted"
exit 1
else
echo "BUG: ci python script is NOT allowlisted"
fi

if printf '%s\n' "$ci_script" | grep -E -v "$non_package_regex" >/dev/null; then
echo "BUG TRIGGERED: grep -v treats ci python script as non-package change"
else
echo "UNEXPECTED: grep -v did not treat ci python script as non-package change"
exit 1
fi
```

## Evidence
- `.buildkite/scripts/common.sh:780-806` defines `non_package_patterns`; it includes `run_dev_scripts_tests.sh` but not `run_ci_python_scripts_tests.sh`.
- `.buildkite/scripts/common.sh:814` uses `grep -E -v "\$\{non_package_regex}"` to classify non-allowlisted changed files as package-affecting.
- `.buildkite/pipeline.yml:63-66` defines the `check-ci-python-scripts` step and executes `.buildkite/scripts/run_ci_python_scripts_tests.sh`.
- Commit `b15f308680aee4377cc27ab82971f19047bb019c` introduced `run_ci_python_scripts_tests.sh`.
- Commit `cc89b23313b6ede1ec4f4e26cf5826f9f4e60290` later added only `run_dev_scripts_tests.sh` to the allowlist, leaving the CI python script path unmatched.

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Bug Hunter](https://github.com/elastic/integrations/actions/runs/26573771271)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Jun 4, 2026, 12:22 PM UTC

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.