lnccbrown / lnccbrown/ssm-simulators

CLI Generate.py ouput folder structure

Open
#172 0 comments 0 reactions 0 assignees View on GitHub
linear-ssm-simulators
Dominant language
Jupyter Notebook
Stars
24
Forks
18
Avg merge
1d 14h
Merged PRs (30d)
9

Description

When one goes to run the `jaxtrain.py` script using the CLI, one of the arguments is `--training-data-folder`, which is the path to the folder with the training data. At the moment, the generated data from `generate.py` is stored within a deep folder structure, for example: `./my_generated_data/data/training_data/lan/training_data_n_samples_20000_dt_0.001/ddm`

Ideally, it would be better for the metadata about the data generation from `generate.py` to be stored in another way, because when writing the CLI command for `jaxtrain` you have to point to the full path for the training data folder (above) rather than the much simpler `./my_generated_data`

One idea is to store the metadata about `n_samples`, `dt`, and the model type in the pickle file name instead.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading generate.py and jaxtrain.py, especially the --training-data-folder handling and the current generated-data layout. Trace how metadata such as n_samples, dt, and model type is written and discovered. Done means the CLI can use a simple generated-data path while retaining enough metadata for jaxtrain.py to locate the training data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.