unreliable test: Q Feature Dev "Iterates on codegen" timeout and accept button failure
- Dominant language
- TypeScript
- Stars
- 2k
- Forks
- 807
- Avg merge
- 10h 12m
- Merged PRs (30d)
- 7
Description
## Test Details
- Name of test: E2E
- VSCode version of failure (minimum/insider/stable): stable
- Link to failing run: https://d1ihu6zq92vp9p.cloudfront.net/175f9cf5-cbfd-4150-aaa7-39128b4084c1/report.html
- Link to failing test: https://us-west-2.console.aws.amazon.com/codesuite/codebuild/135502059251/projects/vscodeE2ETest/history?region=us-west-2&builds-meta=eyJmIjp7InRleHQiOiIiLCJzdGF0dXMiOiIifSwicyI6e30sIm4iOjIwLCJpIjoxfQ
## Log of Test Failure
```
1) Amazon Q Feature Dev
/dev {msg} entry
"before each" hook for "Iterates on codegen":
Error: Test length exceeded max duration: 300 seconds
[No Pending UI Elements Found]
at Timeout._onTimeout (/codebuild/output/src1654629272/src/github.com/aws/aws-toolkit-vscode/packages/core/src/test/setupUtil.ts:46:32)
at listOnTimeout (node:internal/timers:581:17)
at processTimers (node:internal/timers:519:7)
2) Amazon Q Feature Dev
file-level accepts
accept button
button text
shows "Accept all changes" when no files are accepted or rejected, and "Accept remaining changes" otherwise:
AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
assert_1.default.ok(insertCodeButton.pillText === 'Accept remaining changes')
+ expected - actual
-false
+true
at Context. (test/e2e/amazonq/featureDev.test.ts:298:28)
globalSetup: after()
deleteTestTempDirs: deleted 1 test temp dirs
Error: 2 tests failed.
at /codebuild/output/src1654629272/src/github.com/aws/aws-toolkit-vscode/packages/core/src/test/testRunner.ts:128:28
at MultiReporters.done (/codebuild/output/src1654629272/src/github.com/aws/aws-toolkit-vscode/node_modules/mocha-multi-reporters/lib/MultiReporters.js:124:16)
at done (/codebuild/output/src1654629272/src/github.com/aws/aws-toolkit-vscode/node_modules/mocha/lib/mocha.js:1007:16)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
[90m[main 2025-01-29T00:29:31.269Z][0m Extension host with pid 3221 exited with code: 0, signal: unknown.
Exit code: 1
```
## Additional Information
Introduced in: https://github.com/aws/aws-toolkit-vscode/pull/6453
Got the confirmation that this is not an application issue, and partner proceeded with skipping the test: https://github.com/aws/aws-toolkit-vscode/pull/6491.
The skip unblocks our CI and canary. However the issue is not resolved, **a fix is still needed**.
Contributor guide
Research direction
Start with test/e2e/amazonq/featureDev.test.ts around the accept-button assertion and inspect packages/core/src/test/setupUtil.ts at the timeout location. Run the E2E test or review the linked failing run to reproduce both failures. Done means the codegen test completes within the limit and the accept button shows the expected text without skipping the test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript, vscode
- Domain
- devtools, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100