alan-turing-institute / alan-turing-institute/cloudcasting
Discuss train / validation dataset split
- Dominant language
- Jupyter Notebook
- Stars
- 15
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
In the download client we explicitly have he train and validation datasets download to different zarr files. In the pytorch lightning datamodule we have separate dataloaders for train, validation, and test but assuming these are split out in time from the same zarr or set of zarrs. Therefore the download and the datamodule are perhaps assuming different things about how we expect people to be splitting the data to train models. We should think about this and bring these more into harmoney
Contributor guide
Research direction
No files or tests are named. Start by comparing the download client's separate train and validation Zarr outputs with the PyTorch Lightning datamodule's train, validation, and test loaders. Establish the intended split model and align the download and datamodule behavior so their assumptions match.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, pytorch
- Domain
- data, machine-learning
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100