Expected test failures
Nobody has claimed this yet.
- Dominant language
- Racket
- Stars
- 5
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
Sometimes, we expect certain tests to fail specifically when they are run in the build environment.
For example, racket-packet-socket tries to get raw access to an ethernet device, which will clearly fail in the build sandbox. This makes it look like there are failing tests in racket-packet-socket in all the places that the build-server's output is shown to users (e.g. build-server UI itself; package catalog UI), even though that's sort-of not really the case.
It would be nice to have some way of saying either "I expect this and that file to fail its tests in this or that way when run on the build server", or "O build-server, do not run tests for this and that file!"
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how the build server runs tests and how failures are surfaced in the build-server UI and package catalog UI. Use the racket-packet-socket example to understand the expected sandbox failure. Done means the project has a documented way to declare expected failures or skip selected tests without presenting them as ordinary failures.
Written by the indexing model from the issue text.
Assessment
- Domain
- build-system, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100