dart-lang / dart-lang/language
How can we avoid test expectations based on flawed assumptions about internal behaviors?
- Dominant language
- TeX
- Stars
- 2.9k
- Forks
- 239
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 14
Description
See https://github.com/dart-lang/language/issues/2895#issuecomment-1483580690 and the containing thread for some prior discussion.
---
Part of the lifecycle of a language feature is:
1. The language team writes a feature specification.
2. The language team, co19, and others contribute end-to-end tests to ensure that implementations comply with the specification.
3. Implementation teams iterate against the tests in order to implement the feature.
The behaviors mandated in the feature specification vary in how directly they can be expressed in Dart code. If the spec says that the dynamic type of expression `e` is a subtype of `T`, we can test this fairly easily with `e is T` (or better yet, `Expect.type(e)`) - provided we trust that `is` is implemented correctly. On the other hand, if the spec says that the runtime type of `e` is exactly `S`, this is much harder, and we may have to settle for tests that only imply that the implementation is probably correct or tests that only operate over controlled class hierarchies.
In the latter example, a test author might write `e.runtimeType == S`, and while this might work in some cases, it can fail depending on how `.runtimeType` is overridden, and in general, such failures may vary depending on the backend and configuration. When an implementation fails such a test, it can take time to figure out whether the implementation, the test, or even the specification is flawed.
We can't always prevent bad tests from being written, but what can we do to reduce the likelihood of this happening? Some ideas:
* If there's a best way to test a certain behavior, add it to `package:expect`. If a behavior is hard to directly test, maybe make a note of the pitfalls?
* Avoid giving similar names to things that should not be conflated, e.g. "runtime type" and `.runtimeType`. Or if it's unavoidable (as it probably is in this case), make it clear when the two may not be the same thing.
* Many of our tests are derived essentially from a line-by-line reading of the spec. Consider how each line might be tested.
Contributor guide
Research direction
Start by reading the linked dart-lang/language discussion and the containing thread, then review how feature specifications and co19 end-to-end tests express runtime-type behavior. Examine package:expect and existing tests for cases involving `runtimeType`, `is`, and `Expect.type`. Done means producing an agreed, concrete way to reduce tests based on flawed internal assumptions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- dart
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100