aws / aws/aws-toolkit-vscode

unreliable test: Q Feature Dev "Iterates on codegen" timeout and accept button failure

Open
#6,513 0 comments 0 reactions 0 assignees View on GitHub
codewhisperer partner-team tests-ci-cd
Dominant language
TypeScript
Stars
2k
Forks
807
Avg merge
10h 12m
Merged PRs (30d)
7

Description

## Test Details

- Name of test: E2E
- VSCode version of failure (minimum/insider/stable): stable
- Link to failing run: https://d1ihu6zq92vp9p.cloudfront.net/175f9cf5-cbfd-4150-aaa7-39128b4084c1/report.html
- Link to failing test: https://us-west-2.console.aws.amazon.com/codesuite/codebuild/135502059251/projects/vscodeE2ETest/history?region=us-west-2&builds-meta=eyJmIjp7InRleHQiOiIiLCJzdGF0dXMiOiIifSwicyI6e30sIm4iOjIwLCJpIjoxfQ

## Log of Test Failure

```
1) Amazon Q Feature Dev
/dev {msg} entry
"before each" hook for "Iterates on codegen":
Error: Test length exceeded max duration: 300 seconds
[No Pending UI Elements Found]
at Timeout._onTimeout (/codebuild/output/src1654629272/src/github.com/aws/aws-toolkit-vscode/packages/core/src/test/setupUtil.ts:46:32)
at listOnTimeout (node:internal/timers:581:17)
at processTimers (node:internal/timers:519:7)

2) Amazon Q Feature Dev
file-level accepts
accept button
button text
shows "Accept all changes" when no files are accepted or rejected, and "Accept remaining changes" otherwise:

AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

assert_1.default.ok(insertCodeButton.pillText === 'Accept remaining changes')

+ expected - actual

-false
+true

at Context. (test/e2e/amazonq/featureDev.test.ts:298:28)

globalSetup: after()
deleteTestTempDirs: deleted 1 test temp dirs
Error: 2 tests failed.
at /codebuild/output/src1654629272/src/github.com/aws/aws-toolkit-vscode/packages/core/src/test/testRunner.ts:128:28
at MultiReporters.done (/codebuild/output/src1654629272/src/github.com/aws/aws-toolkit-vscode/node_modules/mocha-multi-reporters/lib/MultiReporters.js:124:16)
at done (/codebuild/output/src1654629272/src/github.com/aws/aws-toolkit-vscode/node_modules/mocha/lib/mocha.js:1007:16)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
[main 2025-01-29T00:29:31.269Z] Extension host with pid 3221 exited with code: 0, signal: unknown.
Exit code: 1
```

## Additional Information
Introduced in: https://github.com/aws/aws-toolkit-vscode/pull/6453

Got the confirmation that this is not an application issue, and partner proceeded with skipping the test: https://github.com/aws/aws-toolkit-vscode/pull/6491.

The skip unblocks our CI and canary. However the issue is not resolved, **a fix is still needed**.

Contributor guide

Open the contributing guide

Research direction

Start with test/e2e/amazonq/featureDev.test.ts around the accept-button assertion and inspect packages/core/src/test/setupUtil.ts at the timeout location. Run the E2E test or review the linked failing run to reproduce both failures. Done means the codegen test completes within the limit and the accept button shows the expected text without skipping the test.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript, vscode
Domain
devtools, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.