lincc-frameworks / lincc-frameworks/hyrax
Investigate if we can remove `metadata` from datasets
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 41
- Forks
- 7
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 8
Description
Given that we end up with a HyraxQL (GraphQL-like) method for defining particular aspects of a dataset to utilize during training, inference, and visualization, could we further simplify the dataset classes by treating the metadata tables in a similar way?
The HyraxQL expectation for a dataset is that it will expose a set of getters, such as `get_image`, `get_label`, etc. Currently there is also a `self._metadata_table` that can be used to pull information from. IIRC this was originally implemented for visualization in order to display tabular data when selecting points interactively.
One can imagine a scenario where a metadata table containing say photo-z estimates would be useful for training as well as visualization. So we _could_ automatically generate getters for all the columns that are found in the metadata table, and make those available for use in `to_tensor`.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the dataset classes, the _metadata_table access, and the HyraxQL getters such as get_image and get_label through to_tensor and visualization use. Determine whether metadata-table columns can be exposed as getters without breaking training, inference, or interactive visualization. Done means a documented design decision on whether metadata can be removed and what replacement behavior is required.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100