docker-library / docker-library/ruby
run upstream tests
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 624
- Forks
- 322
- Avg merge
- 22d 3h
- Merged PRs (30d)
- 1
Description
Similar to https://github.com/docker-library/golang/issues/552, there are a lot of upstream tests we could run and should run somewhere, but during build doesn't necessarily make sense.
In https://salsa.debian.org/ruby-team/ruby/-/blob/5b91c2c6b64195da7110d341daba904b9d12c9d5/debian/tests/run-all, we can see that Debian does some clever things to run a lot of the upstream tests post-build via "autopkgtests" (which is very akin to how we run tests on our images after they're built -- they typically run against the built + installed packages).
The basic gist is ruby test/runner.rb, and Debian's got a pretty hefty collection of tests they exclude which will probably inspire tests we need to exclude too (especially # FIXME for now, we are excluding the tests for C extensions; couldn't figure out how to properly build them without building everything else, since we'll have the same issue with doing this post-build).
So we'll need to recreate our "download the source code" logic to get the test files back, but that's the easy part. My own very basic testing of this idea on ruby:slim comes up with "7 failures, 6 errors, 176 skips", so there's clearly some work to do here (for each of those, fixing the image, accommodating the test somehow with runtime flags/packages/etc, or building up a list of tests to skip and writing down justifications for why we think they're valid to skip/ignore).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Debian's debian/tests/run-all and Ruby's test/runner.rb, then compare that post-build approach with the existing Docker image test flow and the reported ruby:slim failures. Done means upstream tests run after the image is built, with failures resolved or excluded and each exclusion documented with a justification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, ruby, shell
- Domain
- devops, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100