Thoughts on command-line vs configuration options

Hi folks,

Currently in https://github.com/pytest-dev/pytest-xdist/issues/789 it has come up that users were asking to be able to configure --dist in the configuration file. While the usual solution is to use addopts for that, in that particular case it was not possible because pytest-xdist errors out if -n is not also passed, which is something you do not necessarily want in your configuration.

That is just an introduction, I'm writing this to put my thoughts on command-line vs configuration options and see what others think (I was writing this as an answer to Ronny in https://github.com/pytest-dev/pytest-xdist/issues/789 but decided to post this here to gather others' options).

Often in pytest we have options which are duplicated, where you have a command-line option which also has a dedicated configuration option for it.

As an example, we have the logging plugin: it has most of its options available in the command-line and also duplicated in the configuration file, which I think creates bloat (cognitive, documentation, code, etc). In many places in that plugin we have to deal with both flavors, and things can go wrong (forgetting to handle both somewhere, changing precedences, etc). IMHO I don't think the logging plugin should be considered an example/guideline on how options should be designed. We need to recall that the logging plugin was created a very long time ago, before pytest even had the -o option, so we inherited all the duplicated configuration.

Currently, users can easily hardcode command-line arguments via addopts, and override any configuration via the -o command-line argument, which I think it is great, and we should leverage that.

In my opinion, we should move our guidlines for options so that an option can only have one of two flavors:

• command-line for options which users frequently change between invocations.
• configuration file for options which are rarely changed, or are really part of the configuration of the test suite and will not change.

I definitely agree that addopts is not as elegant as a dedicated option, but I think it is the worth the tradeoff to avoid bloat.

Having a clear guideline on that would allow us to advertise more about -o (which to this day I'm surprised many people don't know about).

I do not have a strong opinion on this, just wanted to put my thoughts on writing to see if others feel the same, and if is worth having a guideline of sorts on this.

Start by reviewing the linked pytest-xdist issue and the existing discussion about addopts, -o, command-line options, and configuration-file options. Done would require agreement on whether a guideline is warranted and a clearly documented decision; the payload names no specific file or test.