abseil / abseil/abseil-py

absl.flags fails with multiprocessing when using "spawn"

Open
#170 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
279
Avg merge
2d 1h
Merged PRs (30d)
1

Description

There's something odd going on with absl.flags and interacting very badly with multiprocessing when using 'spawn' as a start method. This is on MacOS 11.4, using homebrew's version of Python 3.9.6, although it also fails on the system python 3, 3.8.2.

Given the following code (I'll also atttach it),
[multifail.py.txt](https://github.com/abseil/abseil-py/files/6769112/multifail.py.txt)

"""
import absl.flags
import absl.app
import multiprocessing
import time

absl.flags.DEFINE_integer("delay", 2, "sleep delay")
FLAGS = absl.flags.FLAGS

def worker(n):
time.sleep(FLAGS.delay)
print(n)

def main(argv):
# "fork" works fine.
multiprocessing.set_start_method("spawn")
with multiprocessing.Pool(20) as pool:
pool.map(worker, range(1000))

if __name__ == "__main__":
absl.app.run(main)
"""

it will fail (inconsistently) with

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/homebrew/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/opt/homebrew/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/Users/anthonybaxter/multifail.py", line 10, in worker
time.sleep(FLAGS.delay)
File "/opt/homebrew/lib/python3.9/site-packages/absl/flags/_flagvalues.py", line 499, in __getattr__
raise _exceptions.UnparsedFlagAccessError(error_message)
absl.flags._exceptions.UnparsedFlagAccessError: Trying to access flag --delay before flags were parsed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/anthonybaxter/multifail.py", line 22, in
absl.app.run(main)
File "/opt/homebrew/lib/python3.9/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/opt/homebrew/lib/python3.9/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/Users/anthonybaxter/multifail.py", line 18, in main
pool.map(worker, range(1000))
File "/opt/homebrew/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/opt/homebrew/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
absl.flags._exceptions.UnparsedFlagAccessError: Trying to access flag --delay before flags were parsed.
"""

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.