bahmutov / bahmutov/cypress-split

SPLIT_TIME_THRESHOLD is not respected when using SPLIT_OUTPUT_FILE

Open
#284 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
285
Forks
28
PR merge metrics
No merged PRs in 30d

Description

Wanted to open a discussion about the intended behaviour of `SPLIT_TIME_THRESHOLD` in conjunction with `SPLIT_OUTPUT_FILE`.

**Context**
In my CI setup, the original `timings.json` file is kept in `cypress/e2e//timings.json`, and I am setting the `SPLIT_TIME_THRESHOLD` to be `0.2`.

I am also using `SPLIT_OUTPUT_FILE` to save the updated timings for each parallel node into their own artifacts directory like so:
`artifacts//split-timings/node-<1,2,3,etc>/timings.json`. These are then merged back together by a later merge job using the `cypress-split-merge` binary (which is working great btw!)

**Expected Behaviour (in my head)**
The behaviour I was expecting here was that the `SPLIT_OUTPUT_FILE` would not include updated timings unless the threshold is met. If the threshold was not met, it would either:

1. not write out an updated timings.json at all (this might not work well when merging later)
2. or it would write to the `node-X` output file, but it would be the original times of the test cases and not the updated ones. in this case, when merging the updated timings, Git can realise there is no change to the merged timings file and not bother committing.

**Actual Behaviour**
Despite the changes not meeting the threshold, it writes the updated timings to the `SPLIT_OUTPUT_FILE` anyway:
image

I can see in the code it's because `OUTPUT` !== the `INPUT` file, so we write it anyway: https://github.com/bahmutov/cypress-split/blob/16f7a78c5183c5ccf15cdecab29b0888d368348b/src/index.js#L256

As a result of the above, later in our merge job the spec timings are being updated no matter what the threshold is.

**Question for you**
What would be the ideal way for `SPLIT_OUTPUT_FILE` to respect `SPLIT_TIME_THRESHOLD`? Would love to help hop on a solution!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at src/index.js around line 256 and trace how SPLIT_TIME_THRESHOLD, INPUT, and SPLIT_OUTPUT_FILE affect timing-file writes. Review the cypress-split-merge behavior described in the issue, then determine and test how unchanged timings should be represented in per-node output so later merging does not update timings when the threshold is unmet.

Written by the indexing model from the issue text.

Assessment

Tech stack
cypress, javascript
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.