vitest-dev / vitest-dev/vitest

Feature request: Make "--changed" respect coverage or use static code analysis

Open
#6,735 6 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement: pending triage
Dominant language
TypeScript
Stars
17.1k
Forks
2k
Avg merge
1d 22h
Merged PRs (30d)
94

Description

Clear and concise description of the problem

What if Vitest kept track of which lines each test runs through. That way, the "--changed" flag would be truly only "changed", and could save a huge amount of test execution time.

In our test suite, we have a lot of end-to-end tests, as well as a lot of integration tests. So here, we don't care about Vitest's performance, because that's typically not the limiting factor. IO such as database operations and API calls is limiting instead.

So for that, to really cut off some test execution time, we need to be able to better restrict what tests are run.

--affected does a great job already by looking at the import tree, but it could become even better. If this was implemented, it would be equal to how it works in IntelliJ based IDEs like WebStorm, IntelliJ and Rider, and it could be one of the only test libraries for node that would support this. So there's a lot of opportunity here.

Suggested solution

The way I see it, it could be achieved in three ways.

  1. Using code coverage: Some tools already do this today. However, it may require generating coverage in a specific format which includes this information. Perhaps there's also some performance considerations in basing it on files (so maybe suggestion 2 is better).
  2. Using in-memory code coverage: Same as above, but maybe based on some kind of in-memory data. Vitest must generate existing coverage files today based on some data in memory (that's an assumption), so this could maybe be "easy" to implement directly into the "watch" mechanism of Vite.
  3. Analyzing the AST: Similarly to how imports are analyzed today, we could use the Visitor design pattern or something similar to be able to recursively traverse function calls etc. But that might be more complex, as Node projects can be configured in a variety of ways.
Alternative

No response

Additional context

This was sparked here: https://github.com/vitest-dev/vitest/discussions/6734

Validations

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the existing --affected behavior and the watch mechanism referenced in the issue, then read discussion #6734 for the motivating context. Compare the proposed coverage, in-memory coverage, and AST approaches; the issue does not define a chosen approach or concrete completion criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.