rust-lang / rust-lang/rust

`x test` on profiles where testing is unsupported (or unmaintained?) could set better expectations

Open
#158,385 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-test-infra A-testsuite C-discussion T-bootstrap
Dominant language
Rust
Stars
119k
Forks
16.1k
PR metrics pending

Description

Context

If you run test suites with x.py test (sub)command(s) in environments we don't run it in in CI, they may fail. That's natural and expected, of course, but the tooling output doesn't make it particularly clear what usage is or isn't intended.

If the user is on a weird platform or such, this might be expected anyway; it becomes surprising if the thing that caused a faulure is merely a different bootstrap.toml setup. Most notably if the setup happens to be exactly one of the presets that x.py setup offers.

As an extreme example of the confusion this may lead to: A plain x.py test without any extra arguments, with a basic profile = "dist" profile (created in x.py setup), on a normal x86_64-unknown-linux-gnu (Ubuntu) setup, used to[^1] be prominently displaying an ICE error message as part the first failing test.

  • #158219

An initial search for similar/related kinds of issues had me finding

  • #142505

which seems to also fit this category of problems (though in this case, it's more obviously a case where things should actually just be fixed).

Questions

On one hand, I have some remaining questions about the dist profile: Is the dist profile in particular even meant to be used with commands like x test? I also wonder (i.e. haven't quite figured out yet) what about it makes it more prone to test failures in particular?

On the other hand, I would be interested in judging potential ways forward so that someone who sees a test failure wouldn't even need to wonder about such fundamental things as “is this supposed to work in the first place?”.

👉 expand/collapse the list - [I’m trying not to take away all the focus from the first question]

  1. certain profiles could "disable"[^3] certain x.py subcommands that aren't intended/meant to work anyway, to stop them from running before they even begin

    • this would make the failure quicker, and the error message more consistent than "whatever happens to be the first thing that breaks"
    • the error would provide information on how to change bootstrap.toml, or maybe pass an extra argument, to run the test anyway

  2. or instead, nothing should change - reasons for this may include:

    • there's not really any issue; the test suite is mostly for our internal usage anyway
    • it's too much work to implement and maintain this (for too little benefit)
    • we need better documentation for contributors, and with better documentation, things will be clear enough
    • x test shouldn't fail on dist in the first place because [xyz], we should just fix this

  3. or instead, certain profiles could emit extra warnings/notes with certain x.py subcommands

    • warnings can work similar to out-of-date-change-id warnings, showing both at the start and end of invocation
    • could set expectations through case-specific messages (document known/expected failure; point to context/workarounds for known issues[^2]; or conversely possibly encourage reporting issues for cases that really should work [but not enforced by CI])

  4. or instead, something else? Or a combination of things?

[^1]: at the time of this writing waiting for the merge queue

[^2]: it would ideally be easy to configure certain configuration+command usage patterns, if "known issues" (like ./x test library/std on library profile) should be handled, so contributors can keep them up-to-date easily

[^3]: that is, always produce an error message when it's used

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the x.py setup and x.py test entry points, and inspect how bootstrap.toml profiles—especially dist—affect test invocation. Reproduce the plain x.py test case described in the issue, then determine whether unsupported combinations should be rejected, warned about, documented, or fixed; done requires a decided scope and clear expected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.