webdriverio / webdriverio/webdriverio

Cache list of spec files

Open
#4,880 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted Performance
Dominant language
TypeScript
Stars
9.8k
Forks
2.7k
Avg merge
2d 13h
Merged PRs (30d)
69

Description

Is your feature request related to a problem? Please describe.
Finding specs in a big monorepo takes ages. Think 130K+ files (sources + tests + assets).
Rough directory structure is this:

page1/
	- features/
		page1.feature
	- subpage1/
		- features/
			- subpage1.feature
		- subpage1.js
	- page1.js
	- page1.css
page2/
	<same structure>
ui/shared_component/
	- features/
		- shared_component_in_page1.feature
		- shared_component_in_page2.feature
	- shared_component.js

This is a screenshot from a JS CPU profile of the first ~2 minutes of running WDIO on our repo. At that point, it didn't even get to actually running the tests.

Screen Shot 2019-12-14 at 19 55 39

Note how consistently ConfigParser#getSpecs appears in the chart, and how much time it consumes.

This massive time is to find just ~250 .feature files among the sources.
I also had passed a tag expression that would filter that down to just 4 files, so it's unacceptable that WDIO takes this long.

Describe the solution you'd like
ConfigParser#getSpecs could be memoized, so that when it takes the same input, it always returns the same value.

I guess that adding/removing files could become a problem with that solution, in which case it should be coupled with a file system watching solution to manipulate the cache.

Describe alternatives you've considered
The specs option was too inclusive (src/**/features/*.feature), so I tinkered with the suites option:

suites: {
	page1: ['src/page1/**/features/*.feature', 'src/ui/**/features/*.feature']
	// and so on
}

This didn't help bringing the time to boot up down noticeably, though.

And, as probably obvious from above, keeping a list of .feature files in the config that doesn't make use of wildcards is impossible.

Additional context
Node.js v10.16.3

@wdio packages used:

    "@wdio/cli": "^5.16.10",
    "@wdio/cucumber-framework": "^5.16.10",
    "@wdio/local-runner": "^5.16.10",
    "@wdio/selenium-standalone-service": "^5.16.10",
    "@wdio/spec-reporter": "^5.16.10",
    "@wdio/sync": "^5.16.10",

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ConfigParser#getSpecs in packages/wdio-config/src/lib/ConfigParser.js and review how repeated spec discovery is triggered for large repositories. Define how cached results should respond to added or removed files, then verify that broad glob patterns and tag-filtered feature specs no longer incur the same repeated discovery cost.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
performance, testing-qa, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.