lincc-frameworks / lincc-frameworks/hyrax

Update design: Serialize and allow referencing split indices for reproducibility

Open
#626 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
41
Forks
7
Avg merge
5d 1h
Merged PRs (30d)
8

Description

We are updating the training split design to improve reproducibility and support referencing existing splits for new runs.

## Previous Approach
Splits for train, validation, and test sets were determined on-the-fly based on the provided percentages (or counts), and a random seed was used to achieve reproducibility. However, this could create challenges if new runs needed to reuse exactly the same splits.

## Updated Requirements
- Whenever train/validate/test splits are computed, serialize the sample indices used for each split to the results directory for that run.
- Serialization format should be stable and easily reloadable (e.g., JSON, CSV, or similar), and must record enough information to reconstruct the exact splits.
- On a new training run, the user can reference (point to) a results directory from a prior run in order to reuse those split indices, ensuring true reproducibility between runs.
- If using a referenced index file, the system must verify compatibility (e.g., same dataset, or raise errors if there's a mismatch) and use only the selected indices from those files when creating the splits for the new run.

## Additional Details
- The split indices must be created per data directory/group as needed, and must allow for the complementary/reproducible split logic described previously (i.e., summed split fractions not exceeding 1.0 for a given directory).
- When referencing indices from a previous run, ensure that the splits (train, validate, test) remain complementary and non-overlapping.
- Continue to support both the single-directory and multi-directory split logic from the original proposal.

## Examples / Usage
- After a training run, the results directory might contain:
- `train_indices.json`
- `validate_indices.json`
- `test_indices.json`
- New runs can point to these files to use the exact same data partitions.

All other combination, validation, and "left join" logic from the original design still applies, but now with reproducible split indices as an optional override for future reproducibility.

---
*This update reflects feedback from offline discussion requesting more deterministic and externally referenceable split logic for data handling.*

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or entry points; begin by locating the existing single-directory and multi-directory split computation and results-directory handling. Define the serialization and reference-input interfaces, then verify dataset compatibility, complementary non-overlapping splits, and reproducibility for both supported layouts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.