Multiscene multiprocessing - getting stuck
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 335
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 10
Description
Describe the bug
It is seen that while multiprocessing a function that involves multiscene, some errors are encountered when:
- A scene or multiscene is loaded in the mainthread, before starting the pool.
To Reproduce
import warnings
warnings.filterwarnings("ignore")
from glob import glob
from satpy import Scene, MultiScene
from satpy.multiscene import timeseries
from multiprocessing import Pool
# from satpy.utils import debug_on; debug_on()
filenames = sorted(glob("/home/ubuntu/Roshan/test/data/*.nat"))
bands = ['VIS008']
def process_chunk(filenames_chunk):
mscn = MultiScene.from_files(filenames_chunk, reader='seviri_l1b_native')
mscn.load(bands, upper_right_corner='NE')
blended_scene = mscn.blend(blend_function=timeseries)
print(blended_scene[bands[0]].values.shape)
if __name__ == '__main__':
filenames_chunks = [filenames]
process_chunk(filenames_chunks[0])
p = Pool(processes=1)
results = p.map(process_chunk, filenames_chunks)
p.close()
p.join()
print('out_mp')
Expected behavior
Initially, the process_chunk function is called from the main thread, and output is printed.
When the same function is being called through p.map, it gets stuck.
If the function call from the main thread is commented out, multiprocessing of the function works seamlessly.
Actual results
[DEBUG: 2022-01-24 08:56:27 : satpy.readers.yaml_reader] Reading ('/home/ubuntu/.pyenv/versions/3.9.9/envs/CCNC/lib/python3.9/site-packages/satpy/etc/readers/seviri_l1b_native.yaml',)
[DEBUG: 2022-01-24 08:56:27 : satpy.multiscene] Forcing iteration of generator-like object of Scenes
[DEBUG: 2022-01-24 08:56:27 : satpy.readers.yaml_reader] Reading ('/home/ubuntu/.pyenv/versions/3.9.9/envs/CCNC/lib/python3.9/site-packages/satpy/etc/readers/seviri_l1b_native.yaml',)
[DEBUG: 2022-01-24 08:56:27 : satpy.readers.yaml_reader] Assigning to seviri_l1b_native: ['/home/ubuntu/Roshan/test/data/MSG1-SEVI-MSG15-0100-NA-20211201002742.800000000Z-NA.subset.nat']
[DEBUG: 2022-01-24 08:56:27 : satpy.composites.config_loader] Looking for composites config file seviri.yaml
[DEBUG: 2022-01-24 08:56:27 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2022-01-24 08:56:28 : native_msg] Calibration time 0:00:00.009442
[INFO: 2022-01-24 08:56:28 : satpy.readers.yaml_reader] Flipping Dataset unknown_name upsidedown.
[INFO: 2022-01-24 08:56:28 : satpy.readers.yaml_reader] Flipping Dataset unknown_name leftright.
[DEBUG: 2022-01-24 08:56:28 : satpy.readers.yaml_reader] Reading ('/home/ubuntu/.pyenv/versions/3.9.9/envs/CCNC/lib/python3.9/site-packages/satpy/etc/readers/seviri_l1b_native.yaml',)
[DEBUG: 2022-01-24 08:56:28 : satpy.readers.yaml_reader] Assigning to seviri_l1b_native: ['/home/ubuntu/Roshan/test/data/MSG1-SEVI-MSG15-0100-NA-20211201004242.910000000Z-NA.subset.nat']
[DEBUG: 2022-01-24 08:56:28 : native_msg] Calibration time 0:00:00.009081
[INFO: 2022-01-24 08:56:28 : satpy.readers.yaml_reader] Flipping Dataset unknown_name upsidedown.
[INFO: 2022-01-24 08:56:28 : satpy.readers.yaml_reader] Flipping Dataset unknown_name leftright.
[DEBUG: 2022-01-24 08:56:28 : satpy.readers.yaml_reader] Reading ('/home/ubuntu/.pyenv/versions/3.9.9/envs/CCNC/lib/python3.9/site-packages/satpy/etc/readers/seviri_l1b_native.yaml',)
[DEBUG: 2022-01-24 08:56:28 : satpy.readers.yaml_reader] Assigning to seviri_l1b_native: ['/home/ubuntu/Roshan/test/data/MSG1-SEVI-MSG15-0100-NA-20211201005743.020000000Z-NA.subset.nat']
[DEBUG: 2022-01-24 08:56:28 : native_msg] Calibration time 0:00:00.009205
[INFO: 2022-01-24 08:56:28 : satpy.readers.yaml_reader] Flipping Dataset unknown_name upsidedown.
[INFO: 2022-01-24 08:56:28 : satpy.readers.yaml_reader] Flipping Dataset unknown_name leftright.
(3, 1113, 872)
[DEBUG: 2022-01-24 08:56:28 : satpy.readers.yaml_reader] Reading ('/home/ubuntu/.pyenv/versions/3.9.9/envs/CCNC/lib/python3.9/site-packages/satpy/etc/readers/seviri_l1b_native.yaml',)
[DEBUG: 2022-01-24 08:56:28 : satpy.multiscene] Forcing iteration of generator-like object of Scenes
[DEBUG: 2022-01-24 08:56:28 : satpy.readers.yaml_reader] Reading ('/home/ubuntu/.pyenv/versions/3.9.9/envs/CCNC/lib/python3.9/site-packages/satpy/etc/readers/seviri_l1b_native.yaml',)
[DEBUG: 2022-01-24 08:56:28 : satpy.readers.yaml_reader] Assigning to seviri_l1b_native: ['/home/ubuntu/Roshan/test/data/MSG1-SEVI-MSG15-0100-NA-20211201002742.800000000Z-NA.subset.nat']
[DEBUG: 2022-01-24 08:56:28 : native_msg] Calibration time 0:00:00.009677
Environment Info:
- OS: Linux
- Satpy Version: '0.32.0'
- PyResample Version: '1.22.2'
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the reported reproduction with process_chunk, MultiScene.from_files, mscn.load, and Pool.map, comparing the main-thread and worker paths. Inspect the satpy.multiscene entry points involved in loading and multiprocessing behavior. Done means the same function completes through p.map after the main-thread call.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100