dart-lang / dart-lang/test

Is it possible to have stable IDs across full run/individual test runs?

Open
#837 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Dart
Stars
536
Forks
232
Avg merge
1d 10h
Merged PRs (30d)
10

Description

I'm working on integrating a tree for test results into VS Code:

![screen shot 2018-05-28 at 3 01 58 pm](https://user-images.githubusercontent.com/1078012/40674278-3e4d748c-636c-11e8-9e86-fae521b18bac.png)

This uses the JSON API (via `flutter test`, though eventually it'll handle non-flutter tests). I've added the ability for the user to run individual tests from the tree, but I've hit a problem with matching up nodes in the tree with results that come back which means if the user chooses to run a single test, I end up having to throw away all of the other nodes.

Let's say I run a suite of 10 tests and get back the IDs 1-10. The user then says to run test 5. Since I'm only running by name, it's possible I'll get multiple test results back, but the IDs won't match the previous runs so I can't just update the relevant nodes in the tree (I tried this, but it ended up overwriting node 1 with the new results, leaving a duplicate at node 5).

Is there a way that we could get back matching IDs when we run individual tests (for ex. return the full test list, but only executing the selected test)? I understand the list isn't guaranteed to match the previous one because the user could've changed code, but at least in the case of tweaking and re-running a single test the tree would remain stable.

(Or, if you have any other ideas for handling this, I'm all ears!).

Contributor guide

Open the contributing guide

Research direction

Start by examining the JSON API output from `flutter test` for a full suite and for an individual test run, comparing how result IDs and test names are represented. The issue does not name specific files or tests, so trace the API and test-result entry points before deciding whether stable matching is feasible. Done means an agreed behavior for matching results across runs, or a documented alternative for consumers such as VS Code.

Written by the indexing model from the issue text.

Assessment

Tech stack
dart
Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.