Create ert project concept

The separation between project and runtime settings has been muddy in ERT. For example, why does MAX_RUNTIME — how to run — live in the same configuration as parameters and forward model information — what to run.

I like to compare ERT to git, as there is a similar separation of concerns. Git's workspace concept (ie. where the contents of current HEAD reside — the directory in which .git is found) is vaguely similar to ERT's project directory, where the current project configuration resides. The information about git repositories, git configurations and git hooks are always located within the .ert directory. This means that configuration about the git repository is stored outside of the source code. For example core.excludesFile (optionally, location of additional .gitignore) is found within git's .git, but in ERT such an option would exist within the project's config.ert. The lack of such separation makes it difficult to reason about things like ENSPATH.

The problem with ENSPATH

ENSPATH points to the user's desired storage location, relative to the ERT config, and defaults to storage/. HM tutorial, for example, overrides the default to ../output/storage/<CASE_DIR>.

This is where all the data is saved and in the future this might be a URL to an ERT Storage Server instance rather than a local file path.

Additionally, we want to save the user's configuration at the time of when the user started evaluation their ensembles. This has the benefits of the user being able to tell how specific data has been produced, and potentially reproduce older runs. This config is stored inside of the ERT Storage Server.

Now comes the problem: ENSPATH refers to the location of the storage, and the saved ensembles within the storage also have ENSPATH that is the same. If the user changes their ERT config, we will be able to show them changes between the current and some previous run. However, if the user changes the ENSPATH, all of their data will be "gone", even though we have saved the ENSPATH from previous runs.

This is gross to me. Imagine we had to store the git repository address inside of setup.py [^1].

A potential solution

We can reintroduce .ert to use similarly to .git. The user will be able to set settings via the CLI (ert config set [KEY] [VALUE]), have "remotes" (multiple ERT Storage Server sources?), and other user-specific settings like current_case that can be via CLI or GUI.

This .ert directory will contain information about where the current ENSPATH is (in the form of a remote perhaps). If the user changes the ENSPATH in their ERT config, we'll be able to detect this change and ask the user what to do. Maybe they want to move the data over, or add it as an additional remote, etc.

Additionally, there is a benefit that it will be easy to determine where the data is located. Currently, we have to parse the user's ERT config to determine ENSPATH, which in practice necessarily is done via ResConfig. Having a simple JSON/TOML file that we can read will improve the complexity of our code.

[^1]: setup(..., git_remote="https://github.com/equinor/ert.git"))

Start by reading ResConfig and the existing handling of ENSPATH and config.ert, then trace how project settings are accessed from the CLI and GUI. Clarify the proposed .ert layout, storage remotes, and migration behavior before implementation; done means the project and runtime settings are separated without losing access to existing data.