PytorchConnectomics / PytorchConnectomics/pytc-client
Improvement: Cannot load datasets containing large numbers of 2D files
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
There are two related limitations in the current file loading workflow:
- Cannot select a folder in “Upload from Local”
The file dialog does not allow selecting an entire directory.
While it is possible to select multiple .tif files manually (e.g., using Shift + click), selecting a folder directly is not supported.
- Downstream features only allow single-file selection
For:
- Visualization
- Model training
- Worm error handling
- Other processing steps
Only a single file can be selected at a time.
Impact
When working with datasets that contain thousands of 2D image files, these limitations make it effectively impossible to load and process the dataset efficiently.
Manually selecting thousands of files is not scalable, and loading them one-by-one is not feasible.
Expected behavior
- Allow selecting a directory containing image files.
- Allow loading a folder as a dataset.
- Support batch loading of multiple files across relevant features.
- Allow folder and subfolders to exist under our file management system, so that we can select a whole folder for training image/ask (right now images and masks are paired together in the same folder, so we see one image, one mask, ... This very hard for selecting files (if we get thousands of them) for training/error handling).
This would enable practical workflows for large 2D datasets.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are identified. Begin by tracing the “Upload from Local” selection flow and the selectors used by Visualization, Model training, and Worm error handling; define folder, subfolder, and batch-selection behavior before implementation. Done means large 2D datasets can be selected and processed without one-by-one file selection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- frontend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100