cytomining / cytomining/CytoTable

Loading file from AWS with CytoTable will not show parquet output

Open
#69 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
21
Forks
6
Avg merge
2d 3h
Merged PRs (30d)
6

Description

# Command
```python
import pathlib
import pandas as pd

import cytotable
from parsl.config import Config
from parsl.executors import HighThroughputExecutor

parsl_config = Config(
executors=[
HighThroughputExecutor()
]
)

%%time
what = cytotable.convert(
source_path="/".join(manifest_df.sqlite_file[2].split("/")[0:-1]),
dest_path="test2.parquet",
dest_datatype="parquet",
chunk_size=150000,
parsl_config=parsl_config,
preset="cellprofiler_sqlite_pycytominer"
)
```
# Error

No parquet file is being written to and the `parsl.log` keeps repeating the same output:

```
1686940795.121556 2023-06-16 12:39:55 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.process_loggers:28 wrapped DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-140561780986144
1686940800.118613 2023-06-16 12:40:00 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:164 _general_strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
1686940800.118851 2023-06-16 12:40:00 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:172 _general_strategy DEBUG: Strategizing for executor HighThroughputExecutor
1686940800.119085 2023-06-16 12:40:00 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:193 _general_strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
1686940800.119250 2023-06-16 12:40:00 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:196 _general_strategy DEBUG: Executor HighThroughputExecutor has 1 active tasks, 1/0 running/pending blocks, and 16 connected workers
1686940800.119300 2023-06-16 12:40:00 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:289 _general_strategy DEBUG: Strategy case 3: no changes necessary to current block load
1686940800.119330 2023-06-16 12:40:00 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.process_loggers:28 wrapped DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-140561780986144
1686940805.120214 2023-06-16 12:40:05 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:164 _general_strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
1686940805.120342 2023-06-16 12:40:05 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:172 _general_strategy DEBUG: Strategizing for executor HighThroughputExecutor
1686940805.120624 2023-06-16 12:40:05 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:193 _general_strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
1686940805.120839 2023-06-16 12:40:05 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:196 _general_strategy DEBUG: Executor HighThroughputExecutor has 1 active tasks, 1/0 running/pending blocks, and 16 connected workers
1686940805.120880 2023-06-16 12:40:05 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:289 _general_strategy DEBUG: Strategy case 3: no changes necessary to current block load
1686940805.120915 2023-06-16 12:40:05 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.process_loggers:28 wrapped DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-140561780986144
1686940810.119177 2023-06-16 12:40:10 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:164 _general_strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
1686940810.119357 2023-06-16 12:40:10 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:172 _general_strategy DEBUG: Strategizing for executor HighThroughputExecutor
1686940810.119641 2023-06-16 12:40:10 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:193 _general_strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
1686940810.119845 2023-06-16 12:40:10 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:196 _general_strategy DEBUG: Executor HighThroughputExecutor has 1 active tasks, 1/0 running/pending blocks, and 16 connected workers
1686940810.119901 2023-06-16 12:40:10 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.dataflow.strategy:289 _general_strategy DEBUG: Strategy case 3: no changes necessary to current block load
1686940810.119943 2023-06-16 12:40:10 MainProcess-310074 JobStatusPoller-Timer-Thread-140561780986144-140561778411072 parsl.process_loggers:28 wrapped DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-140561780986144
```
# Details

- Environment is as follows:
```yml
name: jump_sc
channels:
- conda-forge
dependencies:
- conda-forge::python
- conda-forge::boto3
- conda-forge::jupyter
- conda-forge::jupyterlab
- conda-forge::ipykernel
- conda-forge::nb_conda_kernels
- conda-forge::jupyter_contrib_nbextensions
- conda-forge::pandas
- conda-forge::scipy
- conda-forge::numpy
- conda-forge::scikit-learn
- conda-forge::plotnine
- conda-forge::umap-learn
- conda-forge::matplotlib
- pip:
- git+https://github.com/d33bs/cytotable@34d06704c1a9f7db43a13340016f069a6cfd7dbb
- duckdb==0.7.0
```
- Notebook continuously runs without stopping due to the repeated outputs from the parsl.log.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.