E3SM-Project / E3SM-Project/zstash

Debug false success in parallel check/extract

Open
#175 0 comments 0 reactions 1 assignee Claimed by @forsyth2 View on GitHub
Non-reproducible bug
Dominant language
Python
Stars
11
Forks
10
Avg merge
14h 18m
Merged PRs (30d)
1

Description

Initially reported in #167 with partial fix in #168.

`zstash check` was reporting success even if exceptions occurred. This is related to the multiprocessing. Code blocks to investigate would include the following (all in https://github.com/E3SM-Project/zstash/blob/master/zstash/extract.py):

In `multiprocess_extract`:
```
failures: List[FilesRow] = []
while any(p.is_alive() for p in processes):
while not failure_queue.empty():
failures.append(failure_queue.get())
```

In `extractFiles`:
```
except Exception:
# Catch all exceptions here.
traceback.print_exc()
logger.error("Retrieving {}".format(files_row.name))
failures.append(files_row)
```
```
if multiprocess_worker:
# If there are things left to print, print them.
multiprocess_worker.print_all_contents()

# Add the failures to the queue.
# When running with multiprocessing, the function multiprocess_extract()
# that calls this extractFiles() function will return the failures as a list.
for f in failures:
multiprocess_worker.failure_queue.put(f)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.