UMass-Rescue / UMass-Rescue/RescueBox
Rescuebox: deal with large number of files in a folder inputs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6
- Forks
- 15
- Avg merge
- 14h 42m
- Merged PRs (30d)
- 7
Description
when a plugin operation is invoked with input folder that contains a large number of files , it would be better to break it up into chunks and then call the operation, maybe in parallel , and then at the end collate the output.
this could be the steps:
break folders into smaller chunks
filter duplicates and bad files
run operation in parallel and store results
after all chunks are processed summarize results
in case operation stops retry / resume from last processed chunk
how about tying up with pipeline
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how plugin operations receive folder inputs and review the pipeline proposal in issue #56. Define how chunking, duplicate and bad-file filtering, parallel execution, result collation, and retry or resume behavior should fit together. Done means the large-folder workflow is specified and implemented with a way to verify each processing stage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100