Possibility to avoid overriding of argument in `base.context`

Hi @uwefladrich, we area having a discussion within EC-Earth on how to integrate extra configuration features which should not be part of the official repository. Indeed, there are situations where some users/developers would like to modify something from the default that is used within the main context (e.g. model_config, and it is not easy to isolate those modifications and avoid that this does not "pollute" our codebase with a messy when structure. An example could be changing the timestep in certain experiment configuration.

Our current idea is to keep those "configuration" changes in separated files, and in this sense the multiple file approach of ScriptEngine is ideal, i.e. se myconfiguration.yml main.yml. However, the main is coming after and the consequence is that if some option is defined inside the main for general user, is then overwritten and configuration is lost. On the other hand, we cannot call the configuration at the end since the main will take some decision as a function of the configuration option (e.g. the timestep will modify namelists)

After a bit of chatting, one idea it comes out is the possibility to define an option for base.context, for example a directive as !noparse, or even a different command aside of it, e.g base.context_ignore which allow us to circumvent this: if the dictionary key is already defined, the context will ignore the key and keep the original value. This will give the possibility of shipping configuration specific files that are not part of the official command sequence but that can be run as add-ons.

Trying to bring here an example:

Current implementation

When you run se my_config.yml main.yml, the engine processes the files in order. If a key is defined in both, the value in the last file processed takes precedence.

my_config.yml

base.context:
  experiment:
    timestep: 450  # User wants a shorter timestep

main.yml

base.context:
  experiment:
    timestep: 900  # Default value shipped in the repo
    name: "standard_run"
Resulting Context:

The final configuration is:

experiment.timestep: 900 (User's 450 was overwritten by main.yml)
experiment.name: "standard_run"

Desired Behavior:

my_config.yml

base.context:
  experiment:
    timestep: 450  # User wants a shorter timestep

main.yml

base.context_ignore:
  experiment:
    timestep: 900  # Default value shipped in the repo
    name: "standard_run"
Resulting Context:

The final configuration is:

experiment.timestep: 450 #the ignore directive will allow for not overwriting the original value
experiment.name: "standard_run" 

I hope this is clear enough, please ask for clarification if this is not the case. Please notice that I am here to ask for an advice/suggestion on how to implement this, there is already agreement within EC-Earth developers team to develop this potential feature from a fork.

Thanks a lot!

Start at the base.context processing described in the issue and trace how multiple YAML files are combined when later values overlap earlier ones. Define and assess the proposed base.context_ignore behavior against the provided timestep and name examples; done means existing values remain unchanged while new keys are still added.