cytomining / cytomining/CytoTable

`WorkerLost: Task failure due to loss of worker` Failure during join data creation using HTE Parsl config with `SQ00014613.sqlite`

Open
#169 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
21
Forks
6
Avg merge
2d 3h
Merged PRs (30d)
6

Description

This issue occurred while trying to confirm the fix for #38 . After initial data extraction and concatenation (per compartment) CytoTable attempts to join the data in chunks and then concatenates the chunks. Using a Google Colab environment I noticed that I was unable to complete the join chunk creation step when using the HTE (HighThroughputExecutor) from Parsl (a default for CytoTable currently). The error is `WorkerLost: Task failure due to loss of worker #` with little other details on why.

Using a ThreadPoolExecutor for the parsl config results in a successful run within the same environment.

It may be that memory resources are inadequate within Google Colab environments (~12GB) to complete this work with a larger dataset.

Possibly related within the `runinfo/*.err` file. The `parsl.log` file shows the "WorkerLost" error but no other details.
```text
/usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 7 leaked semaphore objects to clean up at shutdown
```

Example [Google Colab notebook](https://colab.research.google.com/drive/1N-klOierrW-c1bWfebMfL2yeEZ3bXxCu) and [backup Gist](https://gist.github.com/d33bs/bb8893f0a100e7074003e49a7652b792).

Contributor guide

Open the contributing guide

Research direction

Start with the linked Google Colab notebook or backup Gist and reproduce join chunk creation for SQ00014613.sqlite using Parsl's HTE configuration, then compare it with the successful ThreadPoolExecutor run. Inspect runinfo/*.err and parsl.log for the worker-loss cause; done means the failure has an actionable explanation and the join either completes or reports the resource problem clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.