E3SM-Project / E3SM-Project/zstash
Debug false success in parallel check/extract
- Dominant language
- Python
- Stars
- 11
- Forks
- 10
- Avg merge
- 14h 18m
- Merged PRs (30d)
- 1
Description
Initially reported in #167 with partial fix in #168.
`zstash check` was reporting success even if exceptions occurred. This is related to the multiprocessing. Code blocks to investigate would include the following (all in https://github.com/E3SM-Project/zstash/blob/master/zstash/extract.py):
In `multiprocess_extract`:
```
failures: List[FilesRow] = []
while any(p.is_alive() for p in processes):
while not failure_queue.empty():
failures.append(failure_queue.get())
```
In `extractFiles`:
```
except Exception:
# Catch all exceptions here.
traceback.print_exc()
logger.error("Retrieving {}".format(files_row.name))
failures.append(files_row)
```
```
if multiprocess_worker:
# If there are things left to print, print them.
multiprocess_worker.print_all_contents()
# Add the failures to the queue.
# When running with multiprocessing, the function multiprocess_extract()
# that calls this extractFiles() function will return the failures as a list.
for f in failures:
multiprocess_worker.failure_queue.put(f)
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.