cytomining / cytomining/CytoTable
`WorkerLost: Task failure due to loss of worker` Failure during join data creation using HTE Parsl config with `SQ00014613.sqlite`
- Dominant language
- Python
- Stars
- 21
- Forks
- 6
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 6
Description
This issue occurred while trying to confirm the fix for #38 . After initial data extraction and concatenation (per compartment) CytoTable attempts to join the data in chunks and then concatenates the chunks. Using a Google Colab environment I noticed that I was unable to complete the join chunk creation step when using the HTE (HighThroughputExecutor) from Parsl (a default for CytoTable currently). The error is `WorkerLost: Task failure due to loss of worker #` with little other details on why.
Using a ThreadPoolExecutor for the parsl config results in a successful run within the same environment.
It may be that memory resources are inadequate within Google Colab environments (~12GB) to complete this work with a larger dataset.
Possibly related within the `runinfo/*.err` file. The `parsl.log` file shows the "WorkerLost" error but no other details.
```text
/usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 7 leaked semaphore objects to clean up at shutdown
```
Example [Google Colab notebook](https://colab.research.google.com/drive/1N-klOierrW-c1bWfebMfL2yeEZ3bXxCu) and [backup Gist](https://gist.github.com/d33bs/bb8893f0a100e7074003e49a7652b792).
Contributor guide
Research direction
Start with the linked Google Colab notebook or backup Gist and reproduce join chunk creation for SQ00014613.sqlite using Parsl's HTE configuration, then compare it with the successful ThreadPoolExecutor run. Inspect runinfo/*.err and parsl.log for the worker-loss cause; done means the failure has an actionable explanation and the join either completes or reports the resource problem clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100