cytomining / cytomining/CytoTable
Loading file from AWS with CytoTable will not show parquet output
- Dominant language
- Python
- Stars
- 21
- Forks
- 6
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 6
Description
# Command
```python
import pathlib
import pandas as pd
import cytotable
from parsl.config import Config
from parsl.executors import HighThroughputExecutor
parsl_config = Config(
executors=[
HighThroughputExecutor()
]
)
%%time
what = cytotable.convert(
source_path="/".join(manifest_df.sqlite_file[2].split("/")[0:-1]),
dest_path="test2.parquet",
dest_datatype="parquet",
chunk_size=150000,
parsl_config=parsl_config,
preset="cellprofiler_sqlite_pycytominer"
)
```
# Error
No parquet file is being written to and the `parsl.log` keeps repeating the same output:
```
1686940795.121556 2023-06-16 12:39:55 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.process_loggers:28 wrapped DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-140561780986144
1686940800.118613 2023-06-16 12:40:00 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:164 _general_strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
1686940800.118851 2023-06-16 12:40:00 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:172 _general_strategy DEBUG: Strategizing for executor HighThroughputExecutor
1686940800.119085 2023-06-16 12:40:00 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:193 _general_strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
1686940800.119250 2023-06-16 12:40:00 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:196 _general_strategy DEBUG: Executor HighThroughputExecutor has 1 active tasks, 1/0 running/pending blocks, and 16 connected workers
1686940800.119300 2023-06-16 12:40:00 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:289 _general_strategy DEBUG: Strategy case 3: no changes necessary to current block load
1686940800.119330 2023-06-16 12:40:00 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.process_loggers:28 wrapped DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-140561780986144
1686940805.120214 2023-06-16 12:40:05 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:164 _general_strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
1686940805.120342 2023-06-16 12:40:05 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:172 _general_strategy DEBUG: Strategizing for executor HighThroughputExecutor
1686940805.120624 2023-06-16 12:40:05 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:193 _general_strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
1686940805.120839 2023-06-16 12:40:05 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:196 _general_strategy DEBUG: Executor HighThroughputExecutor has 1 active tasks, 1/0 running/pending blocks, and 16 connected workers
1686940805.120880 2023-06-16 12:40:05 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:289 _general_strategy DEBUG: Strategy case 3: no changes necessary to current block load
1686940805.120915 2023-06-16 12:40:05 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.process_loggers:28 wrapped DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-140561780986144
1686940810.119177 2023-06-16 12:40:10 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:164 _general_strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
1686940810.119357 2023-06-16 12:40:10 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:172 _general_strategy DEBUG: Strategizing for executor HighThroughputExecutor
1686940810.119641 2023-06-16 12:40:10 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:193 _general_strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
1686940810.119845 2023-06-16 12:40:10 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:196 _general_strategy DEBUG: Executor HighThroughputExecutor has 1 active tasks, 1/0 running/pending blocks, and 16 connected workers
1686940810.119901 2023-06-16 12:40:10 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:289 _general_strategy DEBUG: Strategy case 3: no changes necessary to current block load
1686940810.119943 2023-06-16 12:40:10 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.process_loggers:28 wrapped DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-140561780986144
```
# Details
- Environment is as follows:
```yml
name: jump_sc
channels:
- conda-forge
dependencies:
- conda-forge::python
- conda-forge::boto3
- conda-forge::jupyter
- conda-forge::jupyterlab
- conda-forge::ipykernel
- conda-forge::nb_conda_kernels
- conda-forge::jupyter_contrib_nbextensions
- conda-forge::pandas
- conda-forge::scipy
- conda-forge::numpy
- conda-forge::scikit-learn
- conda-forge::plotnine
- conda-forge::umap-learn
- conda-forge::matplotlib
- pip:
- git+https://github.com/d33bs/cytotable@34d06704c1a9f7db43a13340016f069a6cfd7dbb
- duckdb==0.7.0
```
- Notebook continuously runs without stopping due to the repeated outputs from the parsl.log.
Contributor guide
Assessment
This issue has not been assessed yet.