docker-library / docker-library/ruby

run upstream tests

Open
#503 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
624
Forks
322
Avg merge
22d 3h
Merged PRs (30d)
1

Description

Similar to https://github.com/docker-library/golang/issues/552, there are a lot of upstream tests we could run and should run somewhere, but during build doesn't necessarily make sense.

In https://salsa.debian.org/ruby-team/ruby/-/blob/5b91c2c6b64195da7110d341daba904b9d12c9d5/debian/tests/run-all, we can see that Debian does some clever things to run a lot of the upstream tests post-build via "autopkgtests" (which is very akin to how we run tests on our images after they're built -- they typically run against the built + installed packages).

The basic gist is ruby test/runner.rb, and Debian's got a pretty hefty collection of tests they exclude which will probably inspire tests we need to exclude too (especially # FIXME for now, we are excluding the tests for C extensions; couldn't figure out how to properly build them without building everything else, since we'll have the same issue with doing this post-build).

So we'll need to recreate our "download the source code" logic to get the test files back, but that's the easy part. My own very basic testing of this idea on ruby:slim comes up with "7 failures, 6 errors, 176 skips", so there's clearly some work to do here (for each of those, fixing the image, accommodating the test somehow with runtime flags/packages/etc, or building up a list of tests to skip and writing down justifications for why we think they're valid to skip/ignore).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Debian's debian/tests/run-all and Ruby's test/runner.rb, then compare that post-build approach with the existing Docker image test flow and the reported ruby:slim failures. Done means upstream tests run after the image is built, with failures resolved or excluded and each exclusion documented with a justification.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, ruby, shell
Domain
devops, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.