google / google/ml_collections

Saving ConfigDict to File -- Using ABSL

Open
#19 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1k
Forks
49
PR merge metrics
No merged PRs in 30d

Description

## The Problem:

I'm currently using the following pattern to load a config from dictionary

```python
from absl import app, flags
from ml_collections import config_flags

from .config import MY_CONFIG_DICT

FLAGS = flags.FLAGS

_CONFIG = config_flags.DEFINE_config_from_dict('config', MY_CONFIG_DICT)

# ...
```

`absl` allows me to alter the config from the command line - this is great for running experiments with different parameters. However, if I want to re-use the same configuration for another run, there doesn't seem to be a de-facto way to do this.

For example, I want to be able to run:

```bash
python -m src.experiment --config.value1 10 --config.value2 20
```

and then be able to re-use this config without having to remember to flags.

**How can we save and re-load a `ConfigDict` instance when using `absl` flags?**

## My Approach:

My current approach to this is:

**In the experiment script:**
1. Load config into flags as above.
2. Save the updated `ConfigDict` to a `.yml` file.

**In a postprocessing script:**
1. Load the original config using the same method.
2. Provide an additional flag which takes a path to the saved `.yml` file.
3. Update the original config as follows:

```python
# postprocessing.py

from absl import flags
from ml_collections import config_dict, config_flags

from .config import MY_CONFIG_DICT

FLAGS = flags.FLAGS

_CONFIG = config_flags.DEFINE_config_from_dict('config', MY_CONFIG_DICT)
_LOAD_CONFIG = flags.DEFINE_string('load_config', None, '.yml file from which to update config.')

def update_config():

def _update_config_dict(a, b) -> config_dict.ConfigDict:

da = a.to_dict(preserve_field_references=True)
db = b.to_dict(preserve_field_references=True)

return config_dict.ConfigDict(da | db)

with open(FLAGS.load_config, 'r') as f:
config_from_yaml = yaml.load(f, Loader=yaml.UnsafeLoader)

FLAGS.config = _update_config_dict(FLAGS.config, config_from_yaml)

def main(_):

update_config()

# ...

````

While this works it doesn't seem a great approach.

Is there any standard way to accomplish this - or should I be dealing with updated configs in a different manner?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.