elastic / elastic/ai-github-actions

[bug-hunter] estc-pr-buildkite-detective docs default disagrees with workflow behavior

Open Beginner friendly
#2,070 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11
Forks
16
Avg merge
22h 9m
Merged PRs (30d)
31

Description

## Impact
Users configuring `estc-pr-buildkite-detective` from the published docs/example are told `report-failure-as-issue` defaults to `true`, but the actual workflow default is `false`. In practice, failed Buildkite detective runs do **not** auto-create failure issues unless callers explicitly override the input, reducing failure visibility and breaking expected triage behavior.

## Reproduction Steps
1. Save and run this script from the repository root:

```python
#!/usr/bin/env python3
import pathlib, re, sys

root = pathlib.Path('/home/runner/work/ai-github-actions/ai-github-actions')
wf = (root / '.github/workflows/gh-aw-estc-pr-buildkite-detective.md').read_text(encoding='utf-8')
readme = (root / 'gh-agent-workflows/estc-pr-buildkite-detective/README.md').read_text(encoding='utf-8')
docs = (root / 'docs/workflows/gh-agent-workflows/estc-pr-buildkite-detective.md').read_text(encoding='utf-8')

src_m = re.search(r"report-failure-as-issue:\\n(?:.*\\n){0,6}?\\s*default:\\s*(true|false)", wf)
readme_m = re.search(r"\\| `report-failure-as-issue` \\|[^\\n]*\\| `?(true|false)`? \\|", readme)
docs_m = re.search(r"\\| `report-failure-as-issue` \\|[^\\n]*\\| `?(true|false)`? \\|", docs)

if not (src_m and readme_m and docs_m):
print('FAILED: could not parse one or more defaults')
sys.exit(2)

src = src_m.group(1)
rd = readme_m.group(1)
dc = docs_m.group(1)
print(f'workflow_default={src}')
print(f'readme_default={rd}')
print(f'docs_default={dc}')

if len({src, rd, dc}) != 1:
print('BUG: documented defaults disagree with workflow default')
sys.exit(1)

print('OK: defaults are consistent')
```

2. Run:

```bash
python /tmp/gh-aw/agent/repro-estc-buildkite-default-mismatch.py
```

## Expected vs Actual
**Expected:** Documentation and workflow source agree on the default for `report-failure-as-issue`.

**Actual:** Source default is `false`, while both docs locations say `true`.

Observed output:

```text
workflow_default=false
readme_default=true
docs_default=true
BUG: documented defaults disagree with workflow default
```

## Failing Test
```python
#!/usr/bin/env python3
import pathlib, re, sys

root = pathlib.Path('/home/runner/work/ai-github-actions/ai-github-actions')
wf = (root / '.github/workflows/gh-aw-estc-pr-buildkite-detective.md').read_text(encoding='utf-8')
readme = (root / 'gh-agent-workflows/estc-pr-buildkite-detective/README.md').read_text(encoding='utf-8')
docs = (root / 'docs/workflows/gh-agent-workflows/estc-pr-buildkite-detective.md').read_text(encoding='utf-8')

src_m = re.search(r"report-failure-as-issue:\\n(?:.*\\n){0,6}?\\s*default:\\s*(true|false)", wf)
readme_m = re.search(r"\\| `report-failure-as-issue` \\|[^\\n]*\\| `?(true|false)`? \\|", readme)
docs_m = re.search(r"\\| `report-failure-as-issue` \\|[^\\n]*\\| `?(true|false)`? \\|", docs)
if not (src_m and readme_m and docs_m):
raise SystemExit(2)

src = src_m.group(1)
rd = readme_m.group(1)
dc = docs_m.group(1)
print(src, rd, dc)
raise SystemExit(0 if len({src, rd, dc}) == 1 else 1)
```

## Evidence
- `.github/workflows/gh-aw-estc-pr-buildkite-detective.md` (input definition): `report-failure-as-issue` default is `false`.
- `gh-agent-workflows/estc-pr-buildkite-detective/README.md` (inputs table): default documented as `true`.
- `docs/workflows/gh-agent-workflows/estc-pr-buildkite-detective.md` (inputs table): default documented as `true`.
- `gh-agent-workflows/estc-pr-buildkite-detective/example.yml` does not pass `report-failure-as-issue`, so runtime behavior uses the source default (`false`).

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Trigger Bug Hunter](https://github.com/elastic/ai-github-actions/actions/runs/34593221319)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Sep 18, 2026, 11:27 AM UTC

Contributor guide

Open the contributing guide

Research direction

Read the input definition in .github/workflows/gh-aw-estc-pr-buildkite-detective.md and compare it with the inputs tables in gh-agent-workflows/estc-pr-buildkite-detective/README.md and docs/workflows/gh-agent-workflows/estc-pr-buildkite-detective.md. Run the provided Python consistency check from the repository root. Done means all three reported defaults agree and the check exits successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
90/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.