ContextLab / ContextLab/hypertools

1.1.0: audit follow-ups (gallery repeats, tutorial coverage, native usage)

Open
#284 1 comment 0 reactions 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
1.9k
Forks
164
Avg merge
23h 31m
Merged PRs (30d)
1

Description

Tracking issue for the three audits run on 2026-09-05 before the 1.1.0 re-cut (repeats in the gallery, tutorial coverage of the public API, native-hypertools usage). Everything here ships in 1.1.0. Constraint: the five launch tutorials the Bluesky thread links (`market_sectors`, `weather_decades`, `painting_embeddings`, `conversation_shape`, `morph_shapes_zoo`) keep their pages and clips.

## A. Gallery repeats and misdescriptions (`examples/`)
- [x] Delete `plot_shape_morph.py` (strict subset of `animate_morph_zoo.py`); fold the "all seven shapes" note into `animate_morph_zoo`; retarget `animate_surface_morph.py`'s reference
- [x] Merge `plot_TSNE.py` + `plot_UMAP.py` + `plot_digits.py` into one example
- [x] Merge `plot_PPCA.py` into `plot_missing_data.py`
- [x] Merge `animate_MDS.py` into `animate.py`
- [x] Merge `chemtrails.py` + `precog.py` into one trails page (keep `animate_trails_mix.py`)
- [x] Merge `plot_clusters.py`/`2`/`3` into one three-cell example
- [x] Fix misdescribed docstrings: `plot_story_trajectories.py` (execute the pipeline for real), `plot_hue.py` (categorical hue), `animate_surface_morph.py` (changelog prose), `save_movie.py` (hidden `align=`), `plot_datasets_tour.py` ("five vs four")
- [x] `'hyper'` -> `'HyperAlign'` in `analyze.py`, `plot_align.py`
- [x] Gallery ordering (`within_subsection_order`) and the "Story trajectories" entry in `docs/tutorials.rst`
- [x] Retitle `conversation_trajectories` and `hugging_face_embeddings` so they read as distinct from their siblings

## B. Native usage rewrites
- [x] `load_digits` -> `hyp.load('digits')` (three examples); `load_dataset` -> `hyp.load(...)` (`streaming_data`, `hugging_face_embeddings`)
- [x] `plot_procrustes.py`: private `procrustes()` -> `hyp.align(model='Procrustes')`
- [x] `plot_gensim_text.py`: `text2mat` -> `hyp.plot(texts, vectorizer=..., semantic=...)`
- [x] `plot_autoencoders.py`: `ax.scatter` -> `hyp.plot(..., ax=)`
- [x] `animate_surface_morph.py`: drop the hand downsample; `set_alpha` -> `alpha=` (c4ed79c7 + morph alpha fix)
- [x] Remove dead sentence-transformers / pykalman import guards (six files) (examples done; tutorials in progress) now that extras install on demand
- [x] `projectile_kalman.ipynb` and `stock_forecasting.ipynb` hand `ax.plot` panels -> `hyp.plot(..., reduce=None, ndims=2, ax=)`
- [x] Library gap: isotropic (whole-array) option for `Normalize` (feefcbe9: `Normalize(mode='isotropic')`, 21 tests), so the three morph files stop hand-rolling `normalize_shape()`; tests + docs
- [x] Tighten `tests/test_examples_are_native.py` `DEFECT_MARKERS` with the signatures the audit listed (with a per-file allowlist for deliberate `ax=` demos)

## C. Coverage: new and rebuilt tutorials (each executed, in `docs/tutorials.rst`, install cell, gate entry)
- [x] `hierarchy.ipynb`: column-MultiIndex frame, per-sector traces + mean, `names=`, per-level `alpha=`, hierarchical `predict=` and `return_model` `trace_data`
- [x] `animate_forecast.py` + tutorial: `animate=True, predict='Kalman', forecast_trail=True`, `forecast_hue/palette/fmt`
- [x] `io.ipynb`: `hyp.load` of local files / URL / HF streaming / DataFrame, `hyp.save` round trips, `save_path` formats, `HypertoolsIOError`
- [x] `pipelines.ipynb`: `return_model` on every dispatcher, `.transform`, `pipeline=` replay, `apply_model`, `supported_models()`, bundle keys, `Pipeline` directly
- [x] `manip.ipynb`: `hyp.manip` Smooth/Resample/ZScore/Normalize and chains, `animate='window'` + `focused=`
- [x] Rebuild `plot.ipynb`: saving executed, `animate=`, `names=`, `colorbar=`, `alpha=` list, `size/elev/azim`, labels, `font=`, `label_alpha=`, `backend='plotly'`, `impute=`, `resample=`, exception classes
- [x] Rebuild `align.ipynb`: Procrustes, SRM, `align=` inside `plot`/`analyze`, fitted aligner reuse, local image
- [x] Rebuild `analyze.ipynb`: call every stage it advertises, hypertools figures instead of seaborn heatmaps
- [x] Rebuild `reduce.ipynb`: `describe` curve drawn, UMAP, autoencoders, `return_model`
- [x] Extend `stock_forecasting.ipynb` (Chronos section) and `projectile_kalman.ipynb` (sklearn imputers vs Kalman)

- [x] Library gap found by the morph rewrite: `alpha=` does not reach the travelling morph cloud (matplotlib and plotly); fix + tests, then drop the `set_alpha` call in `animate_surface_morph.py`

## D. Verification and release
- [x] Every tutorial executed with `scripts/execute_tutorial.py`; measured output cells recorded in the gate
- [x] Full suite (4118 passed), ruff, `sphinx -W` gallery build, Colab feature tour (both backends, fresh runtimes)
- [x] Gallery republished from the release commit (3a9b8681); artifacts rebuilt; master pushed; - [ ] CI green; `v1.1.0` moved; tag CI green; draft release updated

### Found while fixing (all fixed for 1.1.0)
- [x] Glyph 8722 warning on caller-supplied `ax=` panels: tick labels under the `sans-serif` alias have no per-glyph fallback -> the `ax=` path sets the tick-label font list (217a2a66)
- [x] `hyp.io.lsl_stream()` left its inlet open -> `LSLStream` destroys it on close/with/GC/exit (61f3e7e8)
- [x] PPCA `divide by zero encountered in log` on wide data -> `slogdet` (54b4b2e2)
- [x] `hyp.load(DataFrame)` raised TypeError -> passthrough for loaded data (a8d073a5)
- [x] `hyp.plot(x, pipeline=)` raised -> fit-or-transform replay of raw steps, `Pipeline.is_fitted` (33405958)
- [x] `DEFECT_MARKERS` tightened with the audit signatures (6b6bcbce); not added: weather's trailing `rolling(12).mean()` (hypertools Smooth is centred; visual rewrites deferred to 1.2 per maintainer)

Also fixed on the way (not in the original audit): HF vectorizers crashing with the default `semantic=` (#285, 1b34b1de), pipeline replay of hand-built Pipelines (33405958), LSL inlet destroy (bb073213), docs-gate literal for the market notebook (d724181b), gensim example title overline, stale gallery output pages.

Contributor guide

Open the contributing guide

Research direction

Start by checking the remaining unchecked release items against the 3a9b8681 release commit and current CI status. Review scripts/execute_tutorial.py and tests/test_examples_are_native.py only as needed to verify the recorded audit results. Done means CI and tag CI are green, v1.1.0 is moved, and the draft release is updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, jupyter-notebook, python
Domain
ci-cd, data-visualization, documentation, release, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.