cytomining / cytomining/CytoTable

Question: Best practices to limit CPU usage?

Open
#437 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
21
Forks
6
Avg merge
2d 3h
Merged PRs (30d)
6

Description

I have been starting some work running CytoTable to work with some data on CellPainting Gallery, and I am wrapping it as part of a Nextflow workflow. One of the things I would like to be able to do is to limit the CPU usage of `cytotable.convert()` while maintaining its efficiency, and I was not quite clear on the best way to do that.

I can limit the number of `parsl` processes/threads with its config, but `duckdb` and `pyarrow` seem to have their own CPU discovery that default to the full set of available cores, so I still end up with quite a large number of threads using that setting alone. It seems like I should be able to also set `CYTOTABLE_MAX_THREADS` and that would limit `duckdb`, but perhaps not `pyarrow`? I think pyarrow can also be limited on its own with `set_cpu_count()` and `set_io_thread_count()`.

So there seem to be a number of possible places to limit the CPU use, and I what I am trying to work out is how best to balance those. Do you have any recommendations for balancing the CPU allocation among `parsl`, `duckdb`, and `pyarrow`?

Secondarily, for a system running within a single Docker container for import, do you have a recommendation between `HighThroughputExecutor` and `ThreadPoolExecutor`? It seems from the paper that multithreading is generally a bit more efficient, but I wanted to confirm that was what you would recommend.

Thanks for your help!

Contributor guide

Open the contributing guide

Research direction

No source file, test, or entry point is identified; start by tracing cytotable.convert() and its Parsl, DuckDB, and PyArrow configuration paths. Done means documenting an agreed CPU-allocation recommendation and the preferred executor for a single Docker container.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
devops, performance
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.