cytomining / cytomining/profiling-handbook

Define folder structures and implement data versioning

Open
#54 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
11
Forks
7
PR merge metrics
No merged PRs in 30d

Description

We want to address two issues here
1. define a new folder structure for profiling experiments
1. identify which of the components will be version controlled.

_I will update this comment periodically as the strategy evolves. I realize this is not ideal because it upsets the chronology of discussions._

This is our [current](https://cytomining.github.io/profiling-handbook/06-appendix.html#directory-structure) folder structure specified in the Profiling Handbook. This differs slightly from the [folder structure](https://github.com/broadinstitute/cellpainting-gallery/blob/main/folder_structure.md) specified in the Cell Painting Gallery. For this level of nesting (under `workspace`) the only discrepancy is `metadata/platemaps` (see #70); `consensus` and `collated` are currently missing in the Gallery, but that is not a discrepancy per se.

This is the proposed folder structure in the Profiling Handbook:

```
├── profiles
│ └── 2016_04_01_a549_48hr_batch1
│ └── SQ00015167
│ ├── SQ00015167_augmented.csv
│ ├── SQ00015167_normalized.csv
│ ├── SQ00015167_normalized_feature_select.csv
│ └── SQ00015167_spherized.csv
├── collated (*)
│ └── 2016_04_01_a549_48hr_batch1
│ ├── 2016_04_01_a549_48hr_batch1_augmented.parquet
│ ├── 2016_04_01_a549_48hr_batch1_normalized.parquet
│ ├── 2016_04_01_a549_48hr_batch1_normalized_feature_select.parquet
│ └── 2016_04_01_a549_48hr_batch1_spherized.parquet
├── consensus (*)
│ └── 2016_04_01_a549_48hr_batch1
│ ├── 2016_04_01_a549_48hr_batch1_augmented.parquet
│ ├── 2016_04_01_a549_48hr_batch1_normalized.parquet
│ └── 2016_04_01_a549_48hr_batch1_spherized.parquet
├── backend
│ └── 2016_04_01_a549_48hr_batch1
│ └── SQ00015167
│ ├── SQ00015167.csv
│ └── SQ00015167.sqlite
├── load_data_csv
│ └── 2016_04_01_a549_48hr_batch1
│ └── SQ00015167
│ ├── load_data.csv
│ └── load_data_with_illum.csv
├── log
├── metadata
│ └── 2016_04_01_a549_48hr_batch1
│ ├── barcode_platemap.csv
│ └── platemap
│ └── C-7161-01-LM6-006.txt
└── pipelines
```

\* collated and consensus files are saved as parquet to allow fast loading.

We will version these folders by placing them inside the project repo

| folder | generator |
| :- | :- |
| profiles | pycytominer|
| collated | pycytominer|
| consensus | pycytominer|
| load_data_csv | pe2loaddata |
| log | GNU parallel (when running various commands) |
| metadata | manual |
| pipelines | manual |

We will not version these folders:

| folder | generator | reason |
| :- | :- | :- |
| backend | cytominer-database|
| analysis | CellProfiler, Distributed-CellProfiler | redundant with SQLite backend |
| images | Microscope | Never changes, and too big! |

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.