frostney / frostney/GocciaScript

Stabilize Test262 CI timeouts for staging recursion cases

Open
#1,071 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug internal performance
Dominant language
Pascal
Stars
20
Forks
3
Avg merge
3d 4h
Merged PRs (30d)
45

Description

Summary

Investigate and stabilize CI-only Test262 timeouts, anchored by the recurring staging/sm/extensions/recursion.js timeout and the additional pass-to-timeout transitions in the latest main artifact.

Why

Retained conformance artifacts should be comparable across commits. A staging test that repeatedly times out on Ubuntu CI but completes quickly on the same source locally makes timeout counts and regression attribution unreliable.

Current behavior

staging/sm/extensions/recursion.js timed out in four consecutive retained Ubuntu main-CI artifacts:

  • run 30950171759 (ed82c755)
  • run 30954990556 (af64b768)
  • run 30962830781 (e95a3e55)
  • run 30980196951 (fb4ca7ae)

The test source is unchanged between the Test262 SHA used for v0.10.0 and the current pin. On current main fb4ca7ae, a local macOS arm64 bytecode run passes the test in about 0.2 seconds with the standard 20-second per-test timeout.

The latest retained artifact reports 13 timeouts instead of 11 in the preceding artifact. These two tests changed from PASS to TIMEOUT even though the intervening product change was parser-only:

  • staging/sm/regress/regress-610026.js
  • staging/sm/TypedArray/element-setting-converts-using-ToNumber.js

The recurring recursion case is recorded at roughly 41 seconds in the artifact, while the other timeout cases are recorded near 20 seconds.

Latest evidence:

Expected behavior

  • Reproduce or instrument the Ubuntu-specific behavior with the same bytecode mode, timeout, memory cap, host-GC support, and profiling configuration as main CI.
  • Determine whether the divergence comes from engine recursion/GC behavior, profiling overhead, worker scheduling, or timeout accounting.
  • Make retained timeout results repeatable across unchanged engine inputs, or explicitly quarantine a demonstrated non-normative environment-sensitive test with a narrow documented contract.
  • Add focused regression coverage for the identified cause without simply increasing the global timeout.

Scope notes

The staging test is non-normative, and current local evidence does not establish an engine correctness failure. Preserve that distinction while investigating CI reproducibility.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/run_test262_suite.ts and test/staging/sm/extensions/recursion.js; reproduce the retained Ubuntu behavior with the same bytecode mode, timeout, memory cap, host-GC support, and profiling configuration. Compare the recursion and pass-to-timeout cases to identify whether engine behavior, profiling, scheduling, or timeout accounting causes the divergence. Done means repeatable results with focused regression coverage, or a narrow documented quarantine contract, without raising the global timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, typescript
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.