lincc-frameworks / lincc-frameworks/hyrax

Custom Metadata Field(s)

Open
#467 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
41
Forks
7
Avg merge
5d 1h
Merged PRs (30d)
8

Description

**Feature request**
When you call the verb `visualize`, the metadata of the dataset catalog is fixed in place (see [Line 139](https://github.com/lincc-frameworks/hyrax/blob/34ca89748b7437c5de15f191a60ea23b403a7c1f/src/hyrax/verbs/visualize.py#L139)) as another `InferenceDataSet` object is called again in the verb `visualize` routine. That means if I have external metadata (e.g., from cross-matching) as a column matched with object ID, I need to append the column to the original dataset metadata table, save it on disk as `fits`, and set the value `astropy_table` in the config to the directory of the saved `fits` file and load it. This way, Hyrax can access the external metadata we appended.

It becomes an issue when we want to modify the appended column and visualize it on an interactive UMAP. When I try to plot with a color column on the interactive UMAP, I have to re-train the entire dataset since there is no way Hyrax can read the external metadata after training if I don't set `astropy_table` beforehand (because the verb `visualize` creates another `InferenceDataSet` and uses it). Moreover, the metadata is independent of training results, so training won't affect the external information anyway. Instead of just modifying the column on the fly, I have to train the dataset again, and it doesn't do anything with the training. Therefore, I think it might be good to have some customizability in including external metadata after training (e.g., during the interactive UMAP visualization).

Please let me know if I've missed something or if you have any questions.

**Before submitting**
Please check the following:

- [ ] I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
- [ ] I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
- [ ] If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/hyrax/verbs/visualize.py at line 139 and trace how the visualize routine creates another InferenceDataSet and reads the astropy_table configuration. Clarify the intended interface for supplying or modifying external metadata after training, then identify tests covering interactive UMAP visualization. Done should allow external metadata to be used for coloring an interactive UMAP without retraining, with an agreed acceptance test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-visualization, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.