google / google/grain

How are config options meant to be set in Grain?

Open
#1,360 0 comments 0 reactions 0 assignees View on GitHub
type:support
Dominant language
Python
Stars
779
Forks
86
Avg merge
2d 6h
Merged PRs (30d)
6

Description

Having faced the issue described in #1355, I was reading through the codebase to identify the source of the invalid flag read. While the read causing this specific issue seems to be [here](https://github.com/google/grain/blob/d2568262e43ef1e43a7400e59caba2b2f3e1c35f/grain/_src/core/profiler.py#L69), the pattern of flag usage here as well as in the config module confuses me a little.

While every config option has an associated ABSL flag, my understanding is that, as per standard usage, flags need to be parsed in order to be accessed within the application (using `absl.flags.FLAGS(argv)`). The workaround mentioned in the above issue (using `absl.flags.FLAGS.mark_as_parsed()`) appears to be a workaround that does not align with standard flag usage.

Further, I noticed two other anomalies -
1. I noticed that certain flags are manually "set" as defined during application runtime, which appears as another workaround for similar issues (ex: [here](https://github.com/google/grain/blob/0b1f2f24871d2311b517d7a284d369f91477ad4e/grain/_src/python/dataset/transformations/process_prefetch.py#L73))
2. None of the config options appear to be set when using `grain.config.update(, )`, again because the underlying flag is not set unless the flags are parsed

**Question**: Is there some documentation on how these options are meant to be used with Grain? Are library users expected to parse the flags at the point of usage, and is there a reason for this design? Let me know if I am misunderstanding the control flow here. Thanks!

Contributor guide

Open the contributing guide

Research direction

Start by reading grain/_src/core/profiler.py and grain/_src/python/dataset/transformations/process_prefetch.py, then trace the config module and its flag parsing behavior. Document how library users should set options, whether flags must be parsed, and what grain.config.update is expected to do; done means the intended usage and rationale are clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
developer-experience
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.