Fine grained locking in storage
- Dominant language
- Python
- Stars
- 161
- Forks
- 141
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 138
Description
Implement some way for several process to access the same experiment in storage.
This would make it easier to split ert into several processes.
The scheduler can not currently be split out of ert due to the internalization of run path after each iteration.
The gui needs at least read access to an experiment.
Forward model updates also modify storage.
Splitting ert into several processes would help with GIL contention issues such as what we experienced when running many realizations at the same time. Splitting ert into several processes may also help define clearer interfaces between ert components.
It would also be possible to move internalization of run path into the job runner if we can do fine grained locking.
If we do this then each job runner must import ert.storage which can take several seconds. This is mostly an issue for unit tests, and can probably be solved by using forkserver with preloaded imports. See https://bnikolic.co.uk/blog/python/parallelism/2019/11/13/python-forkserver-preload.html
The upside of moving it to job runner is that we parallelize the import over all the job nodes instead of doing all the imports in scheduler after a full iteration is completed.
One solution might be to implement per file locking.
Contributor guide
Research direction
Start by reading ert.storage and tracing how the scheduler, GUI, forward model updates, and job runner access experiment storage. The issue proposes per-file locking but does not select a design or name files or tests; done requires a decided fine-grained locking approach that supports the described concurrent processes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100