Immutable metadata

Desired capability or behavior.

Metadata is a simple dictionary, and as such can be freely modified at any point. This is actually desirable in the form of extending or merging metadata, but this can also cause problems:

1. Metadata gets copied or saved in several locations, such as passed as options to OpenMDAO components, arguments to functions or methods, or saved as attributes in objects. We want to make sure these are all identical for a given AviaryProblem. Component writers also shouldn't be able to modify metadata (although if they do and mess things up its easy to assign them the blame...)
2. If not carefully copied before being used, metadata that is changed can still be tied to the same reference as other python variables when the user did not intend that (or realized that happened). This can cause very hard-to-diagnose issues, especially in the test suite, where tests that modify metadata accidentally get applied to another test that thought it was just using the unmodified CoreMetaData. This is dependent on test execution order, so when running tests in parallel the test that fails can seem "random" and not happen consistently between test runs.

While we are modifying metadata, we should consider streamlining the name. Right now we use "metadata", "meta_data"/"meta-data", "MetaData", etc. and should be consistent. In code we should use a consistent variable name, and then in documentation we should pick a specific natural English spelling to refer to the concept (and use the code/variable name when specifically referring to code. I prefer "metadata" as a single word.

Suggested implementation.

We should consider making metadata an immutable object, or have some way of toggling immutability (i.e. once AviaryProblem has loaded it in). AviaryProblem needs to be able to modify metadata, as it merges the metadata given to it with the metadata of each external subsystem.

One solution is a immutable MetaData class that, once initialized (perhaps with the existing _MetaData dict), cannot be changed. To allow for metadata merging, any manipulation of the MetaData class (such as adding, removing, changing variables) returns a new copy of the object, similar to how NumPy arrays work.

The variable hierarchy has the same problem as metadata - could they possibly be combined into a single object? This would have some benefits and other substantial downsides, but having users need to be responsible for creating two separate and almost redundant variable files is clumsy.

Pydantic??

This may be the place where we start using Pydantic if we decide that is the way we want to go for Aviary. This might make sense to wait for OM4 and also gather any other places we want to use Pydantic, such as a phase_info replacement.

Start by tracing how AviaryProblem loads and merges metadata, including the existing _MetaData dict, and review how the variable hierarchy is represented. Compare immutable-object and Pydantic options, considering the possible OM4 timing mentioned in the issue. Done should include a decided metadata API that supports AviaryProblem's merges, prevents unintended mutation afterward, and establishes consistent naming.