[test] Better test failure reporting
Open
Nobody has claimed this yet.
- Dominant language
- Ruby
- Stars
- 2.2k
- Forks
- 256
- Avg merge
- 6d 17h
- Merged PRs (30d)
- 37
Description
If you have a test which fails, the stdlib test reports like the following:
Minitest::Assertion: Expected ["[Dir.mktmpdir] ReturnTypeError: expected `::Integer` but returns `\"/var/folders/b9/sw1f2b996cd_tclwhhmh5tfc0000gn/T/d20200108-67558-4ri1a4\"`", "[Dir.mktmpdir] ReturnTypeError: expected `::Integer` but returns `\"/var/folders/b9/sw1f2b996cd_tclwhhmh5tfc0000gn/T/foo20200108-67558-2jmzq0bar\"`", "[Dir.mktmpdir] ReturnTypeError: expected `::Integer` but returns `\"/var/folders/b9/sw1f2b996cd_tclwhhmh5tfc0000gn/T/foo20200108-67558-315pdj\"`", "[Dir.mktmpdir] ReturnTypeError: expected `::Integer` but returns `\"/var/folders/b9/sw1f2b996cd_tclwhhmh5tfc0000gn/T/d20200108-67558-42bd5d\"`"] to be empty.
It tells us there is an inconsistent method call but does not tell which one is the source of the failure.
We need an improvement to help to identify the source of the failure.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported failure and reading the stdlib test failure output shown in the issue. Trace how the inconsistent method calls are reported, then make the failure identify which call is the source and verify that the example output becomes more actionable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100