NatLabRockies / NatLabRockies/gaps

Add `batch` option to not copy all files in top-level directory

Closed
#41 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement p-medium topic-batch
Dominant language
Python
Stars
1
Forks
10
Avg merge
1d 4h
Merged PRs (30d)
8

Description

Why this feature is necessary:
It's inefficient (and potentially problematic) to copy all files form the top-level directory into the sub-directories, especially is a careless user stores a large file in the main directory.

A possible solution is:
Add a key to the batch config that allows the user to set a copy behavior. The default can still be 'all' (though maybe this is worth reconsidering). However, a new option can be "config" or something similar, which reads all the config files in the pipeline config and only copies files that the config "mentions". This is not perfect, since there can be a files that a user wants to be copied into the sub-directories that does not appear in any config files, but this is a rather fringe example which can still be handled by the original "all"
copy method. The "config" method is likely sufficient for most typical use-cases.

Urgency / Timeframe
Not super urgent, just a nice-to-have

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the batch configuration handling and the logic that copies files from the top-level directory into sub-directories. Review how pipeline config files are read; done means preserving the default 'all' behavior while adding an option that copies only files mentioned by the configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.