nf-core / nf-core/deepmodeloptim
[future][discussion] deal with data located in files
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Nextflow
- Stars
- 31
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
Description of feature
Current implementation requires a flat-file input csv file where one line = one entry (see readme ).
However, most datasets are not flat, i.e. image datasets require data to be saved in external files
We should consider :
- the shuffle process shuffles the lines of the input csv file, image datasets have few files that each contain lots of information (which would decrease the effectiveness of shuffle). idem for the split method.
- how to mount those files to the various processes ?
- effective way of parsing those files
- memory allocation for large datasets
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the flat-file input contract in the README, then trace how the pipeline handles CSV shuffling and splitting. Review the issues raised around external files, process mounting, parsing, and memory allocation. Done would require an agreed design for supporting non-flat datasets, rather than a single localized change.
Written by the indexing model from the issue text.
Assessment
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100