PytorchConnectomics / PytorchConnectomics/pytc-client

Improvement: Cannot load datasets containing large numbers of 2D files

Open
#124 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2
Forks
6
PR merge metrics
No merged PRs in 30d

Description

There are two related limitations in the current file loading workflow:

  1. Cannot select a folder in “Upload from Local”

The file dialog does not allow selecting an entire directory.
While it is possible to select multiple .tif files manually (e.g., using Shift + click), selecting a folder directly is not supported.

Image Image
  1. Downstream features only allow single-file selection

For:

  • Visualization
  • Model training
  • Worm error handling
  • Other processing steps
    Only a single file can be selected at a time.
Image

Impact

When working with datasets that contain thousands of 2D image files, these limitations make it effectively impossible to load and process the dataset efficiently.

Manually selecting thousands of files is not scalable, and loading them one-by-one is not feasible.

Expected behavior

  • Allow selecting a directory containing image files.
  • Allow loading a folder as a dataset.
  • Support batch loading of multiple files across relevant features.
  • Allow folder and subfolders to exist under our file management system, so that we can select a whole folder for training image/ask (right now images and masks are paired together in the same folder, so we see one image, one mask, ... This very hard for selecting files (if we get thousands of them) for training/error handling).

This would enable practical workflows for large 2D datasets.

Image

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are identified. Begin by tracing the “Upload from Local” selection flow and the selectors used by Visualization, Model training, and Worm error handling; define folder, subfolder, and batch-selection behavior before implementation. Done means large 2D datasets can be selected and processed without one-by-one file selection.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.