ContextLab / ContextLab/hypertools

Fold-in candidates from the 1.1 examples/tutorials audit (native features to add)

Open
#285 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.9k
Forks
164
Avg merge
23h 31m
Merged PRs (30d)
1

Description

On 2026-09-05 three read-only audits went through every hand-written (non-`hyp.*`) block in the 51 `examples/*.py` scripts, the 25 `docs/tutorials/*.ipynb` notebooks (code cells; install cells skipped), the 53 code cells of `notes/colab/hypertools_1.1_feature_tour.ipynb`, and the earlier native-usage audit's "no equivalent" (C/P) items, asking which of them are worth folding into the library as native features. Each block was checked against the current tree (`inspect.signature(hyp.plot)`, the `plot.py` docstrings, `matplotlib_backend.py`, `animate.py`, `colors.py`, `text2mat.py`, `io/sources.py`, the `manip` registry) and, where a claim was cheap to test, run in `.venv`. The consolidated, de-duplicated list is below. Constraint for the whole list: the five launch clips (`market_sectors`, `weather_decades`, `painting_embeddings`, `conversation_shape`, `morph_shapes_zoo`) must stay visually identical in 1.1, so anything that changes those clips, or requires rewriting those examples, is 1.2 work. An item is marked `1.1 candidate` only when it is effort S, touches no launch clip, and removes a trap or a line of boilerplate that recurs; everything else is `1.2` by default. Within each group items are ordered by recurrence, then effort.

## Bugs and traps found

- [x] **HF vectorizers need `semantic=None, corpus=None`** (DONE for 1.1.0: 1b34b1de -- embedding vectorizers skip the semantic stage by default; four tutorials simplified) — `text2mat.py:355-364` auto-skips the topic stage only for gensim names, so every Hugging Face / sentence-transformers call spells out the triple (`vectorizer='BAAI/bge-small-en-v1.5', semantic=None, corpus=None`). Proposed: extend the auto-skip in `hypertools/tools/text2mat.py` to the HF tier (a name not in the sklearn or gensim registries), warning only when the user passed a topic model explicitly. Effort S. Recurs: 15 cells in 4 notebooks (wikipedia_embeddings 5, 6, 10, 11; hugging_face_embeddings 4-6; conversation_trajectories 7-8; conversation_shape) plus `animate_painting_embeddings.py` and `animate_conversation.py`. Plotly: n/a (data stage). Verdict: **1.1 candidate** — S, the two launch text examples keep passing the triple explicitly so their clips are byte-identical, and it removes the most-repeated boilerplate line in the docs (and the silent trap of an LDA stage running on top of embeddings).
- [x] **`reduce=None, ndims=2` rescales into the unit box and loses axis units** — measured: every 2-D plot is mean-centred, rescaled into `[-1, 1]` and pinned to `xlim/ylim=(-1.1, 1.1)`, with `ax=` too and on plotly (a series with x = 0..94 came back with xdata in [-1.0, 0.68]; `matplotlib_backend.py:2476-2477, 2636-2637`, docstring `plot.py:1341`). So `hyp.plot(..., reduce=None, ndims=2, ax=ax)` is NOT a drop-in for the hand `ax.plot` time-series panels (years on x, degrees on y); the previous audit's P5 "convert now" is re-opened. Proposed: `axis_scale='data'` (or `frame=False`) on 2-D static plots: raw coordinates, real ticks, no frame square, in `matplotlib_backend` 2-D draw + `plotly_backend`. Effort M. Recurs: weather L282/288-289, projectile_kalman 7, 8, 10, stock_forecasting 8, 10, tour cell 43 (7 sites). Plotly: same rescale measured, same fix needed. Verdict: 1.2 (opt-in; weather adopts in 1.2). **DONE 1.1.0: axis-units group (`axis_scale='data'`, `xlim=`/`ylim=`, `ndims=1` series mode, `truth=`, `predict=[...]`)**
- [x] **HF progress bars / tokenizer env vars must be set before import** — three notebooks and two scripts set `HF_HUB_DISABLE_PROGRESS_BARS`, `HF_HUB_VERBOSITY`, `TOKENIZERS_PARALLELISM` by hand, with the comment that they "must be set before importing the libraries that read them, so these imports are intentionally not at the top of the cell" (`grep -rn HF_HUB_DISABLE hypertools/` finds nothing). Proposed: `os.environ.setdefault(...)` immediately before the lazy `sentence_transformers` import in `hypertools/tools/text2mat.py::_hf_fallback_model` (~L126), or `hyp.config['quiet_hf'] = True`. Effort S. Recurs: wikipedia_embeddings 2, hugging_face_embeddings 2, conversation_trajectories 2, `scripts/execute_tutorial.py:59`, `scripts/render_bluesky_clips.py` (5 sites). Plotly: n/a. Verdict: **1.1 candidate** — S, logging-only (no pixels), removes three identical notebook preambles and two script hacks. **DONE 1.1.0: 7e424516 (text2mat) + 7d50f21d (sources)**
- [x] **`ndims=1` drops the index, rescales y, and refuses 2+ columns** — measured: `hyp.plot(np.array([5,7,6,9,8.]), reduce=None, ndims=1)` draws with x = 0..N (a DataFrame's float index named `t` was discarded), y rescaled to [-1, 1] (first vertex `[0.0, -1.0]`), and a 2-column frame raises `ValueError: ... static plots support at most 1`. So the tutorials build `np.column_stack([t, y])` panels with `reduce=None, ndims=2` instead. Proposed: an honest `ndims=1` time-series mode in `plot.py` + both backends: with `reduce=None` draw each column against the DataFrame index (or `x=`), raw units, honouring `xlabel/ylabel`. Effort M. Recurs: stock_forecasting 8, 10; projectile_kalman 7, 8, 10 (5 sites). Plotly: same path, per-column traces. Verdict: 1.2 (a documented behaviour change; pairs with `axis_scale='data'` and the `truth=` overlay below). **DONE 1.1.0: axis-units group (`axis_scale='data'`, `xlim=`/`ylim=`, `ndims=1` series mode, `truth=`, `predict=[...]`)**
- [x] **`font=` never reaches per-segment titles; title size/weight only come from rcParams** — `plot.py:7745-7747`: `resolve_font` "only ever sets a FontProperties `family=`, never a `size=`", and "for a per-segment `title=` list specifically, `_make_title_updater` sets the title with NO `fontproperties=` override at all" (`plot.py:8060-8079`: `axes.set_title(titles[min(idx, len(titles) - 1)])`), which resets size/weight/colour/y every frame. Proposed: pass the resolved `font=` through `_make_title_updater`, and carry size/weight via `title_kwargs=` (Plotting, below). Effort S. Recurs: `animate_morph_zoo.py:175-184` (`restyle_title`), `animate_conversation.py:228-241`, `animate_market_sectors.py:280` (`rc_context`) — 3 launch files. Plotly: `layout.title.font` already carries family/size. Verdict: 1.2 — any change on the title path needs a frame-by-frame pixel diff of three clips. **DONE 1.1.0: 17fd2f7f**
- [x] **Multi-line title reservation** — an animated 3-D plot grows its figure by ONE measured title-line height (`_animated_3d_title_line_height_in`, probe `'Xygj'`, `plot.py:7710`); measured: `title='two\nlines'` gives the same 5.04 in figure height as a one-line title, so the second line runs off the canvas. That is exactly what `make_room_for_title` (`animate_conversation.py:244-267`, mirrored in conversation_shape) works around. Proposed: in `_reserve_animated_3d_title_margin` (~L7840) reserve `n_lines * line_height + pad` with `n_lines = title.count('\n') + 1` (max over a per-segment list), honouring `title_kwargs['fontsize']`. Effort S. Recurs: 2 sites (conversation example + notebook). Plotly: no reservation needed. Verdict: 1.2 — a bug fix, but it CHANGES conversation_shape's clip (the example would double-reserve until `make_room_for_title` is removed); land both together, or in 1.1.x with that one clip re-rendered deliberately. **DONE 1.1.0: 5f69f369 (conversation example drops its own helper; clip re-render flagged for review)**
- [x] **`_load_hf` drops ClassLabel names** — `hypertools/io/sources.py:696-708` returns `ds.to_pandas()` with no ClassLabel decoding; hugging_face_embeddings cell 3 works around it ("a streaming load keeps the dataset's own label names; a plain (non-streaming) load returns a DataFrame with integer labels instead") via `news.features['label'].names`. Proposed: decode `ClassLabel` columns with `ds.features[col].int2str` before `to_pandas()`, opt-out `decode_labels=False`. Effort S. Recurs: 1 site. Plotly: n/a. Verdict: **1.1 candidate** — S, no clip involvement, removes a quoted library trap (note it is a return-value change for non-streaming HF loads; the opt-out and a CHANGELOG line cover it — Jeremy's call). **DONE 1.1.0: 7d50f21d + 595a878a (`decode_labels=`)**
- [x] **Bundled Noto Sans ships only a Regular face, so `fontweight='bold'` silently falls back** — quoted from morph_shapes_zoo / `animate_morph_zoo.py` ("hypertools' bundled default (Noto Sans) ships only a Regular face, so `fontweight='bold'` alone silently falls back to regular"), which is why the zoo restyles with DejaVu Sans. Proposed: document it in the `font=` docstring and warn when a weight is requested with the bundled family (shipping a Bold face is the alternative). Effort S. Recurs: 1 site. Plotly: n/a. Verdict: 1.2 (fold into the `title_kwargs=` work). **DONE 1.1.0: 67d08df4**

## Plotting

- [x] **Multi-panel static plots: `panels=` / `subplots=` / `hyp.subplots`** — the strongest recurrence in the tree: `plt.subplots(nrows, ncols, subplot_kw={'projection': '3d'})`, `ravel()`, loop `hyp.plot(x, ax=ax, title=..., show=False)`, hide spares, `tight_layout()`. `inspect.signature(hyp.plot)` has no `panels`/`grid`/`subplots`/`ncols`; `ax=` is the only composition route. Proposed: `hyp.plot(list_of_datasets, panels=True | int | (rows, cols) | list_of_titles)` (aliases `subplots=`), one static axes per dataset or per `reduce=` entry when `reduce` is a list (`reduce=['PCA', 'UMAP', 'Isomap']` = one panel per reducer), `title=` one string per panel, spares hidden, one Figure returned; smaller fallback `fig, axes = hyp.subplots(nrows, ncols, ndims=3, size=...)`; keep the `animate=` refusal (`plot.py:2845`). In `plot.py` + `matplotlib_backend._draw`. Effort M (mpl) + M (plotly parity via `make_subplots(specs=[[{'type': 'scene'}]*n])`; today `ax=` under plotly only appends to a figure). Recurs: 6 examples (plot_datasaurus 27-43, plot_shapes_zoo 26-42, plot_datasets_tour 56-68, plot_autoencoders 53-61, plot_gensim_text 41-57, plot_impute 53-57) + 7 tutorial sites in 5 notebooks (plot 23; align 3, 8, 11; reduce 10; stock_forecasting 10; projectile_kalman 7) + tour cells 32, 66 = 15 sites. Verdict: 1.2 (opt-in, static only, no clip is static; too big for the freeze). **DONE 1.1.0: 17fd2f7f (`panels=`, `subplots=`, `hyp.subplots`)**
- [x] **Title styling: `title_kwargs=` + per-segment `title_color=`** — three launch examples carry an `on_frame` whose only job is to re-apply size/weight/family/colour/y after the library's updater resets them (`speaker_title`, `date_title`'s style half, `restyle_title`), and market wraps the call in `plt.rc_context({'axes.titlesize', 'axes.titleweight'})` so the margin probe sizes for the bigger font. Proposed: `title_kwargs={'fontsize': 24, 'fontweight': 'bold', 'fontfamily': ..., 'y': 0.93, 'color': ...}` applied by the library's own setter (static block in `matplotlib_backend` ~L2719 and `_make_title_updater`), values optionally per-dataset lists for serial `title=` lists, plus `title_color=[...]` or a callable `ctx -> colour` for the per-frame tint; feed fontsize to the `_reserve_animated_3d_title_margin` probe (retires the `rc_context`). Effort S (dict) / M (per-segment colour). Recurs: `animate_market_sectors.py:280, 295-304`, `animate_conversation.py:228-241`, `animate_morph_zoo.py:175-184`, `animate_weather_decades.py:297-298` (4 launch files). Plotly: `layout.title.font` (size/color/family) is direct. Verdict: 1.2 (opt-in kwarg, but adopting it in three clips needs pixel diffs; the examples audit: "ship the kwarg in 1.1 only if the clips are re-rendered and compared; otherwise 1.2"). **DONE 1.1.0: 17fd2f7f**
- [x] **Per-dataset `hue=` broadcast** — `hue=[[speaker] * len(windows) for ...]` repeats one label per row so datasets that share a speaker share a colour. Proposed: let `hue=` accept one scalar per DATASET (`len(hue) == len(x)`, entries not sequences) and broadcast over rows: `hue=speakers`. Effort S. Recurs: conversation_trajectories 6, conversation_shape (`hue=[[speaker] * len(turn) ...]`), text 5, 6 (4 sites). Plotly: hue resolution is shared upstream of the backends. Verdict: 1.2 (S, but the single-dataset case where `len(hue) == n_rows` is ambiguous; needs a rule and tests before it ships). **DONE 1.1.0: 17fd2f7f**
- [x] **Per-dataset `labels=` with `label_anchor=`** — both users build an all-`None` label list with one string per dataset to annotate a dataset once (painting on the MIDDLE observation, plot_labels on the FIRST point). Proposed: `labels=` of length n_datasets annotates each dataset once, `label_anchor='first'|'center'|'last'`. Effort S. Recurs: `animate_painting_embeddings.py:259-261`, `plot_labels.py:22-30` (2). Plotly: annotations already drawn by both backends. Verdict: 1.2 (opt-in; painting's clip only changes if adopted — `center` must reproduce "middle window" exactly). **DONE 1.1.0: 17fd2f7f**
- [x] **Legend under mixture / matrix hue + `legend_kwargs=` / `legend_colors=`** — `plot.py:5466-5468 / 5483-5485`: "legend is not supported for continuous or matrix-valued hue; ignoring legend", so market hand-builds six `Line2D` handles plus a grey `'#666666'` "Market" entry. Proposed: in the `hue_is_matrix` branches build proxy handles from `palette` (one entry per hue-matrix column) instead of discarding; `legend_kwargs={'loc': 'upper left', 'fontsize': 8, 'frameon': False}` and per-entry `legend_colors=[...]` (or `(label, color)` tuples) for the bespoke market swatch, `matplotlib_backend` legend block (~L2739). Effort S. Recurs: `animate_market_sectors.py:288-293` (1; the only mixture-hue example). Plotly: `showlegend` dummy traces. Verdict: 1.2 (library side is 1.1-safe, but it has one user, the warning already tells them, and adopting it changes the market clip). **DONE 1.1.0: 17fd2f7f**
- [x] **Expose the resolved colour scale: `bundle['colors']` / `HyperAnimation.colors`** — weather rebuilds `Normalize(mean.min(), mean.max())` + `plt.get_cmap('RdBu_r')` to reproduce the library's hue mapping in its companion panels; `_build_colorbar_info` computes `vmin/vmax/palette` (`plot.py:7377-7378`) but the `return_model` bundle exposes only `fig, xform_data, trace_data, trace_metadata, animation, pipeline, models, predict` (`plot.py:3159-3160`). Proposed: `bundle['colors'] = {'kind': 'continuous', 'cmap', 'norm', 'vmin', 'vmax', 'palette'}` (categorical: `'colors'` + `BoundaryNorm`) built from `colorbar_info` at bundle assembly (~L7083), also `HyperAnimation.colors`. Effort S. Recurs: `animate_weather_decades.py:241-253` (1; painting re-derives colours by a different mechanism). Plotly: same dict from the plotly colorbar info. Verdict: 1.2 (additive and clip-safe, but only weather uses it and it does not remove a trap; take it with the weather 1.2 rewrite). **DONE 1.1.0: 17fd2f7f (`bundle['colors']`, `HyperAnimation.colors`)**
- [x] **`palette=` dict keyed by hue category** — conversation computes a first-appearance category order by hand so a `palette=` list lines up with the speakers. Proposed: `palette={'Alice': '#E4572E', 'Bob': ...}` in `colors.py`. Effort S. Recurs: `animate_conversation.py:276` (1). Plotly: shared palette resolution. Verdict: 1.2. **DONE 1.1.0: b2b37c91 + 17fd2f7f**
- [x] **Per-dataset image palettes + `image_palette(..., max_luminance=)`** — `palette=` is one string/list for the whole plot (`colors.py:297`, single `IMAGE_PALETTE_PREFIX` parse), and `image_palette(image, n_colors, resize, random_state)` (`colors.py:330`) has no luminance floor, so painting picks the first colour with luminance <= 0.6 by hand (`canvas_color`, L185-202) and passes `color=`. Proposed: `palette=['image:a.jpg', 'image:b.jpg', ...]` one entry per dataset -> first salient colour each, and `image_palette(..., max_luminance=None)` passed through from `palette='image:...'`. Effort S. Recurs: `animate_painting_embeddings.py:185-202` (1). Plotly: palette resolution is backend-neutral. Verdict: 1.2 (painting's colours come out identical only if the luminance floor is exposed too; otherwise keep `color=`). **DONE 1.1.0: b2b37c91 + 17fd2f7f**
- [x] **`title_wrap=`** — conversation runs `textwrap.fill` over its titles. Proposed: `title_wrap=int` on `plot()`. Effort S. Recurs: `animate_conversation.py:279-280` (1). Plotly: `
` insertion. Verdict: 1.2 (with the multi-line reservation fix; on its own it makes the second line run off the canvas). **DONE 1.1.0: 17fd2f7f (multi-line reservation fix in the animation wave)**

## Animation

- [x] **Reveal progress on parallel / spin / window animations (`ctx.progress`, populated `revealed_counts`)** — both time-indexed clips reinvent `frac = ctx.frame / (n_frames - 1)` and interpolate into the row index to find "where the head is" ("A parallel reveal exposes no reveal count", `animate_weather_decades.py:293-294`); `matplotlib_backend.py:1265, 1336, 1564, 2175, 2285` build `FrameContext(... revealed_counts=None)` on the non-serial paths. Proposed: populate `ctx.revealed_counts` (or add `ctx.progress` float) on those paths, in `animation_context.FrameContext` + both backends. Effort S. Recurs: `animate_market_sectors.py:298-300`, `animate_weather_decades.py:295-296` (2). Plotly: same context object. Verdict: 1.2 (additive metadata and neither market nor weather reads `revealed_counts`, so clips are safe, but it does not remove a trap or a boilerplate line until the hooks are rewritten). **DONE 1.1.0: 5f69f369 (`ctx.progress`, `window_bounds`, populated `revealed_counts`)**
- [x] **`title=` from a callable or a DatetimeIndex pattern** — market and weather format a date into the title every frame from the row calendar. Proposed: `title=` accepting a callable `ctx -> str`, and for DataFrame input with a `DatetimeIndex` a strftime pattern such as `title='{index:%B %Y}'`, in `plot.py` (`_make_title_updater`). Effort M. Recurs: `animate_market_sectors.py:295-304`, `animate_weather_decades.py:297-298` (2). Plotly: per-frame `layout.title`. Verdict: 1.2 (adoption changes the two clips only if the formatted string differs; still a rewrite of launch hooks). **DONE 1.1.0: 5f69f369**
- [x] **Morph loop closure: `loop=True` for `animate='morph'`** — "The sampling IS done by hand rather than left to morph_samples: the loop-closing repeat of the first cloud has to be the SAME sample, and morph_samples draws a fresh subset per dataset" (`animate_morph_zoo.py:63-65`, sampling at L98-115). Proposed: `loop=True` appends a hold on the first cloud reusing its sampled points, or make `morph_samples` reuse the sample for an identical (`is`/equal) repeated array, in `morph.sample_and_match_clouds` + the `matplotlib_backend` morph path (plotly morph shares the sampler). Effort S. Recurs: morph zoo (1). Plotly: shared sampler. Verdict: 1.2 (dropping the hand `rng.choice` changes the sampled point set -> re-render). **DONE 1.1.0: 5f69f369 (`loop=True`)**
- [x] **`HyperAnimation.drawn_extent(frames=None)`** — painting measures the union bbox of everything drawn over the orbit from rendered pixels (`figure_box` / `drawn_extent`, L226-250, 280-283) to place its prose and thumbnail columns beside the spinning cube; the helper is already generic. Proposed: a method on `HyperAnimation` (`animate.py`) returning the bbox in figure fractions. Effort S. Recurs: painting (1). Plotly: n/a (no pixel layout). Verdict: 1.2 (no trap, one user). **DONE 1.1.0: 5f69f369**
- [x] **Serial cross-dataset recency fade: `dataset_fade={'floor', 'decay'}`** — "``chemtrails``/``precog``/``bullettime`` fade WITHIN one trajectory; nothing in 1.1 fades ACROSS already-revealed datasets" (`animate_conversation.py:189-191`); the 55-line `recency_fade` hook sets per-dataset alpha `FLOOR + (1-FLOOR)*DECAY**k` from `current_index`, splitting heads/trails in `ctx.artists` ("``ctx.artists`` is NOT one artist per dataset. It is heads first, then trails"). Proposed: `hyp.plot(..., animate='serial', dataset_fade={'floor': 0.15, 'decay': 0.7})` (or `recency='exponential'`) in `animate.py` + `FrameContext`; dataset i drawn at `floor + (1-floor)*decay**(current-i)`, current at 1.0, unrevealed at 0, applied to head AND trail (the library's 0.3x trail convention differs, so the kwarg must set both to the same alpha to keep the clip identical). Effort M. Recurs: conversation (1). Plotly: per-frame opacity per trace, feasible. Verdict: 1.2 (opt-in; conversation adopts in 1.2 with a pixel compare). **DONE 1.1.0: 5f69f369 (`dataset_fade=`)**
- [x] **Linked animated companion panels** — weather's revealed `LineCollection` time-series panel, rolling mean and head marker (L275-290) plus the on_frame that updates it; `ax=` + `animate=` raises (`plot.py:4095`) and the `ax` doc says "Several animated panels in one figure are not supported; lay the panels out in the DATA instead". Proposed: `hyp.plot(..., animate=True, linked_axes=[(line_ax, 'reveal')])` (or `companion='timeseries'`) revealing a 2-D companion trace on the same frame schedule, in `matplotlib_backend` + `animate.py`; depends on `axis_scale='data'`. Effort L. Recurs: weather (1). Plotly: a second `xy` subplot with per-frame data, separate work. Verdict: 1.2+ (new subsystem). **DONE 1.1.0: 5f69f369 (`companion=`, matplotlib; plotly raises)**

## Text

- [x] **Sliding text windows: `hyp.tools.text_windows` / `window=`** — three different chunkers turn one document into a trajectory: word windows with a min-windows guard (`animate_painting_embeddings.py:149-153`, `animate_conversation.py:137-150`, conversation_shape), equal-length character chunks (`chunk(s, count)`, text 3), and 3-sentence regex windows (`split_sentences`, conversation_trajectories 5). Proposed: `hyp.tools.text_windows(text, size, step, unit='chars'|'words'|'sentences', min_windows=)` and a `window=(size, step)` option on the text path (`format_data`/`text2mat`, exposed on `plot()`) so a single string becomes one observation per window. Effort M. Recurs: 5 sites. Plotly: n/a (data stage). Verdict: 1.2 (no appearance change if the same size/step are used, but adopting it rewrites two launch examples). **DONE 1.1.0: 74334d8a `hyp.text_windows` (plot `window=` deferred to the plot wave)**
- [x] **`hyp.load('wiki')` / `hyp.load('nips')` as lists of strings** — the hosted text corpora come back as a `(3136, 1)` object array (`load.py:315-319` documents the shape), so three sites do `[str(p) for p in x[0].ravel()]`; `hyp.load('sotus')` already returns strings. Proposed: return a list of strings, with a CHANGELOG compat note. Effort S. Recurs: text 7, 8; wikipedia_embeddings 3 (3). Plotly: n/a. Verdict: 1.2 (a return-type change on a public loader; wants a deprecation line rather than a freeze-week flip). **DONE 1.1.0: f60dadd2**
- See also, filed under Bugs and traps: HF vectorizers defaulting `semantic`/`corpus` to None, and the HF progress-bar env vars.

## Data sources and synthetic data

- [x] **Synthetic dataset family: `hyp.load('random_walk' | 'helix' | 'blobs' | 'moons' | 'swiss_roll' | 's_curve' | 'lorenz', **kwargs)`** — there is NO generated dataset in `sources.py` (everything is downloaded or bundled; `SKLEARN_DATASETS` covers only `load_*`, `sources.py:184-210`), so the tour defines `walk()` (called 20x), `spiral3d()`, `blobs()`, and the docs hand-roll random walks in 10 files (tour, plot.ipynb, streaming_data, plot_pipelines_return_model, plot_nested_lists, plot_apply_model, plot_multicolored_lines, plot_missing_data, plot_interactive_backend, animate_plotly), helices in 3 (tour, plot_predict, plot_multiindex), blobs in 2 (tour, modern_sklearn_dynamics), moons and Lorenz via `solve_ivp` (modern_sklearn_dynamics 5; streaming_data 3's `live_feed` is an Euler-stepped Lorenz in 10-D). Proposed: new `hypertools/io/synthetic.py` dispatched from `sources.py` after `EXAMPLE_DATA` and before the sklearn table; deterministic under `random_state=`, `n_datasets=` returns a list, `hyp.load('lorenz', streaming=True, dim=10)` yields the generator streaming_data hand-rolls; `blobs`/`moons`/`swiss_roll`/`s_curve` are `sklearn.datasets.make_*` passthroughs, `lorenz` uses scipy (core dep). Effort S. Recurs: ~17 generators in ~16 files. Plotly: n/a. Verdict: 1.2 (S per dataset but a new public surface with docs in io.rst; no clip involvement — the launch examples' `synthetic_weather/market/climate/shapes` fallbacks stay where they are, they are shaped like the real feeds). **DONE 1.1.0: 7d50f21d + 595a878a**
- [x] **Hierarchy frame builder `hyp.tools.stack` (subsumes an `aggregate='mean'` trace)** — MultiIndex frames are built by hand with `pd.concat({('listeners', f'subject {i}'): df}, axis=1)` + `columns.names = [...]` (`multiindex.py` exposes only `expand_multiindex` / `build_multiindex_styles`), and group means are computed with `np.mean(list_slice, 0)` and drawn as extra traces even though a hierarchy frame already draws "leaves plus derived per-level means" (plot.py `legend` doc). Proposed: `hyp.tools.stack({'group 1': arrays, 'group 2': arrays}, names=['Group', 'Subject', 'Feature'], keys=..., axis=1)` (or `hyp.io.hierarchy`) returning the frame plot/predict already understand; the align tutorial's `group_averages` disappears. Effort M. Recurs: hierarchy 3, 10, 11; `plot_multiindex.py`; `animate_market_sectors.py` (frame) + L269 (`np.mean(aligned, axis=0)`); align 5-7, 11; analyze 11; `plot_align.py:25-26`; `save_movie.py:35-36` (~10 sites). Plotly: n/a. Verdict: 1.2. **DONE 1.1.0: 74334d8a**
- [x] **URL download cache: `hyp.load(url, cache=True, offline=True)`** — `hyp.load(url)` has no on-disk cache for arbitrary URLs (only built-in datasets under `~/hypertools_data`; `grep -n cache sources.py` shows only per-process seaborn/538 listing caches), so each fetcher implements download + User-Agent + atomic `.part` write + offline fallback. Proposed: opt-in cache under `~/hypertools_data/urls/` and an `offline=True` mode that only reads the cache, in `io/sources.py`. Effort S-M. Recurs: animate_forecast 3 (`urllib` CSV), stock_forecasting 2 (yfinance snapshot), projectile_kalman 2 (GitHub 7z), weather `_cached_download`, market cache (5). Plotly: n/a. Verdict: 1.2. **DONE 1.1.0: 7d50f21d + 595a878a (`cache=`, `offline=`, HypertoolsOfflineError)**
- [x] **Web sources: `hyp.load('wikipedia:')`, `hyp.load('yahoo:<TICKER>', start=, end=)`, `hyp.load('sec:<TICKER>', concept=)`** — `grep -i wikipedia\|yfinance\|yahoo\|edgar\|sec hypertools/io/` finds only the `wiki` topic-model note; `sources.py` resolves example / sklearn / seaborn / fivethirtyeight / kaggle / local / HF / Sheets / Dropbox / URL only. Today: text 3 (raw MediaWiki `extracts` via `requests` + User-Agent + try/except fallback to inline snippets, "live Wikipedia fetch failed ... falling back"), wikipedia_embeddings 8 (`wikipediaapi` keyword loop with `page.exists()` skip), stock_forecasting 2 (`yf.download` with 3 retries + CSV snapshot), market `fetch_prices` (Yahoo v8 chart, L145) / `fetch_shares` (SEC `company_tickers.json` + XBRL `companyconcept`/`companyfacts`, L94-95, 170). Proposed: explicit-prefix sources in `io/sources.py` (+ `sources_web.py`) behind `lazy_import`/`requests`, with the cache dir above; Wikipedia returns the plain-text extract (`exintro=1` option, list form for several titles); Yahoo returns daily OHLCV with a `DatetimeIndex` (always send period1/period2 — `range=max` silently returns 3-month bars, per `.claude/CLAUDE.md`); SEC needs a User-Agent with a contact and `companyconcept` can be empty for a filer whose `companyfacts` has the concept. Effort M (wikipedia) / M (yahoo) / L (sec). Recurs: 5 sites (a `convokit/<name>` source was considered for conversation_trajectories and left — the filtering is the tutorial's point). Plotly: n/a. Verdict: 1.2 (a source landing does not change pixels — the clips are built from `fixture_data()`/cached JSON — but rewriting the examples onto it is 1.2). **DONE 1.1.0: 7d50f21d**
- [x] **`hyp.io.lsl.synthetic_outlet(name, n_channels=6, rate=100)`** — lsl_streaming 3's `start_synthetic_outlet()` (a real `pylsl.StreamOutlet` on a daemon thread pushing sinusoids) duplicates `tests/test_lsl_streaming.py:97` `_sample_for_index`. Proposed: a helper in `hypertools/io/lsl.py` returning `(thread, stop_event)`. Effort S. Recurs: 2 (tutorial + test). Plotly: n/a. Verdict: 1.2 (low priority). **DONE 1.1.0: 3800012c**
- [x] **Tour cell 88: hosted `cube` / `sphere` instead of hand-made clouds (optional, no library change)** — `hyp.load('cube')` and `hyp.load('sphere')` are hosted `npz_array` shapes (`load.py:43-44`) and `morph_samples=300` caps them; `ring` has no hosted analogue. `hyp.plot([hyp.load('cube'), hyp.load('sphere'), ring], animate='morph', morph_samples=300)` works today but costs two downloads in a section that is deliberately offline. Effort S. Recurs: 1. Verdict: optional convert (tour only; not a 1.1 blocker). **DONE 1.1.0: tour refreshed (local notebook)**

## Forecasting and imputation

- [x] **Backtest / model-comparison scoring for `predict` and `impute`** — hand-rolled hold-out splits, naive last-value baselines, MAE/MAPE/RMSE rows (per axis, occluded-only), pivot tables and a best-vs-naive verdict; the `predict` docs also show `hyp.predict(model=...)` has no list form (`predict/predict.py:160-230`). Proposed: `hyp.predict(data, model=['Kalman', 'ARIMA', ...], t=30, holdout=True, metrics=('mae', 'mape'), baseline='naive')` and `hyp.impute(damaged, model=['Kalman', 'KNNImputer', 'IterativeImputer'], truth=full)` returning a tidy scores frame (model x dataset x metric, scattered vs row-gaps) beside the forecasts/fills, describe-style `show=True` bar chart; `predict/common.py` + `impute/common.py` over a shared `core/evaluate.py`, kwarg-gated so the default path is untouched. Effort M-L. Recurs: stock_forecasting 4-7; projectile_kalman 6, 8, 9; tour cell 41 (7 sites — the tour audit judged that one "scoring scaffolding; no hypertools scoring util and none warranted", the tutorials audit disagrees). Plotly: bar chart parity only. Verdict: 1.2. **DONE 1.1.0: 79da0fa4**
- [x] **`hyp.tools.damage(x, frac=, rows=, seed=)`** — four impute demos knock out cells/rows by hand (toeplitz walk + 5 % `rng.choice`; `rng.random(shape) < 0.05`; occlusion of 5 full rows + 10 % scattered NaNs with a `.flat` write-through workaround: ".to_numpy() can return a read-only, Fortran-ordered array; a plain .ravel() on that silently returns a *copy* ... so writes through it would vanish"). Proposed: a helper returning the damaged copy + mask (owning the pandas pitfall). Effort S. Recurs: plot 21, 22; pipelines 9; projectile_kalman 5; tour 41 (5). Plotly: n/a. Verdict: 1.2 (new public function; the trap it removes is pandas', not ours). **DONE 1.1.0: 74334d8a**
- [x] **`truth=` forecast overlay** — stock 8 / projectile 10 draw train-tail, held-out and forecast as three `np.column_stack([day, price])` datasets with `['-', '-o', '--x']`. Proposed: `hyp.plot(train, predict='Chronos', t=30, truth=held_out)` draws the actual continuation beside the forecast in the same space (static and animate), in `plot.py` + both backends; builds on the honest `ndims=1` mode above. Effort M. Recurs: stock_forecasting 8, 10; projectile_kalman 7, 8, 10 (5). Plotly: extra trace with the forecast palette. Verdict: 1.2. **DONE 1.1.0: axis-units group (`axis_scale='data'`, `xlim=`/`ylim=`, `ndims=1` series mode, `truth=`, `predict=[...]`)**
- [x] **Multi-model forecast overlay: `predict=['Kalman', 'ARIMA', 'GP']`** — tour cell 43 wants several forecasters on one trace and falls back to `ax.plot(signal[:,0],'k')` + one `ax.plot(f.index, f.iloc[:,0])` per model; `predict=` takes one spec. Proposed: a list form, one overlay per model coloured from `forecast_palette`, `legend=True`, in `plot.py` + `predict/predict.py`. Effort M. Recurs: tour 43 (1). Plotly: one trace per model. Verdict: 1.2. **DONE 1.1.0: axis-units group (`axis_scale='data'`, `xlim=`/`ylim=`, `ndims=1` series mode, `truth=`, `predict=[...]`)**

### Adjacent time-series / alignment tools

- [x] **`Smooth(..., align='trailing', min_periods=)`** — `manip/smooth.py:14` `KERNELS = ('savgol', 'gaussian', 'boxcar')` are all centred (no `trailing`/`causal`/`center` option), so weather keeps `pd.Series(mean).rolling(12).mean()` (L275). Proposed: `hyp.manip(x, model='Smooth', kernel='boxcar', kernel_width=12, align='trailing')` with pandas rolling semantics (NaN for the first k-1 rows or `min_periods=`). Effort S. Recurs: weather (1). Plotly: n/a. Verdict: 1.2 (opt-in; the only item a launch example could adopt with byte-identical output, so it could ride a 1.1.x — but it replaces one pandas line, not a trap). **DONE 1.1.0: 2812e4a8 as `Smooth(center=False, min_periods=)` (align= collides with the cross-module align stage)**
- [x] **Alignment quality score: `hyp.align(..., return_score=True)` / `hyp.describe(data, aligned=...)`** — `plot_story_trajectories.py:65-77` computes `dispersion()` (mean distance to the cross-dataset centroid / cloud scale) before vs after alignment. Proposed: a dispersion/ISC-style score returned from `align` or reported by `describe`. Effort S. Recurs: 1. Plotly: n/a. Verdict: 1.2. **DONE 1.1.0: 98b4d2ff**
- [x] **Time-delay embedding manipulator `manip='Delay'`** — modern_sklearn_dynamics 6 builds a Takens embedding (`tau=5, dims=20`) with `np.column_stack`; the `hyp.manip` registry (`manip/manip.py:32`) is `[Normalize, ZScore, Smooth, Resample]`, and Kalman already delay-embeds internally (`predict/kalman.py:75`). Proposed: `hypertools/manip/delay.py` (`hyp.manip(x, model='Delay', lags=20, tau=5)`, usable as `manip='Delay'` inside plot), sharing Kalman's helper. Effort S-M. Recurs: 1. Plotly: n/a. Verdict: 1.2. **DONE 1.1.0: 2812e4a8**

## Already closed in 1.1

- **Isotropic Normalize** (tour/gaps report): `hypertools/manip/normalize.py:8` `MODES = ('minmax', 'isotropic')`; both surviving morph files use it (`animate_morph_zoo.py:109`, `animate_surface_morph.py:60`, morph_shapes_zoo.ipynb); `examples/plot_shape_morph.py` was deleted in 78568adc (gallery de-duplication), so all three P2 copies are gone. The examples report independently marks `animate_surface_morph.py` fully native.
- **Morph `alpha=`** (tour/gaps report): `hypertools/plot/morph.py:383` `morph_alpha` ("GH #284: alpha= reaches the traveling cloud").
- **Guards / `load_digits` markers (P1 / P3)** (tour/gaps report): HEAD 6b6bcbce added six DEFECT_MARKERS to `tests/test_examples_are_native.py:204-210` (`\bload_digits\b`, `load_dataset(`, `text2mat(`, `align.procrustes`, `import sentence_transformers`, `find_spec(`); `grep -rl` over examples/ and docs/tutorials/ finds no remaining `import sentence_transformers`, `find_spec('sentence_transformers')` or `load_digits(`.
- **Fully native already** (examples report): `animate_surface_morph.py` (`hyp.manip` normalize, rotations list, `surface=` dict) and `animate_forecast.py` (`predict=`, `forecast_*`, `slow_warning_seconds=`) need nothing; 30 further scripts are single or sequential `hyp.*` calls. The tutorials report likewise leaves cluster 9 / analyze 10 as intentional demos of `hyp.cluster` / the returned Pipeline (not equivalent to `plot(cluster=...)`, which clusters the reduced scores, `plot.py:2115`).
- **Load passthrough, pipeline replay, LSLStream**: named as closed in the earlier native-usage audit; none of the three reports consolidated here re-checked them, so they are carried forward without fresh evidence.

## "Convert now" items (tutorial/example edits, no library change)

- text.ipynb cell 5: `hue = ['dog'] * len(dog) + ['cat'] * len(cat); hyp.plot(dog + cat, 'o', hue=hue, size=[8, 6])` -> `hyp.plot([dog, cat], 'o', legend=['dog', 'cat'], size=[8, 6])` (cell 6 likewise for the hue half; it still needs a per-point `labels=` list). **DONE (708838a9; re-executed)**
- animate_forecast.ipynb cell 3: the `urllib`/`.part`/`pd.read_csv(dest)` fetch -> `hyp.load(ARCHIVE)` (io.ipynb cell 6 already loads the sibling `temperature_locs.csv` from the same repo that way); keep the region selection, `dropna().tail(N_MONTHS)` and the synthetic fallback. Note this notebook mirrors `examples/animate_forecast.py`, so the change belongs in the script first. **DONE in part (708838a9): the cached CSV is read through `hyp.load(dest)`; the download + `.part` cache stays until the URL-cache fold-in above lands.**
- analyze.ipynb cell 10: `per_dataset = np.split(...)` is computed and never used -- delete. **DONE (708838a9; re-executed)**

Tracked from #284 (the 1.1 audit); the three "convert now" items and any `1.1 candidate` marked done below ship in 1.1.0, everything else is 1.2 planning.

**2026-09-05: every item above is implemented for 1.1.0.** Launch examples were left visually unchanged except `animate_conversation`, whose own title-room helper double-reserved once the library reserved every title line (helper removed; clip re-rendered 12 px shorter, content identical). Follow-ups for 1.2: convert the launch examples (market, weather, painting, morph zoo) to the new kwargs once their clips can change; a `window=` kwarg on `hyp.plot` wrapping `hyp.text_windows`; `panels=` for animations beyond `companion=`.

Contributor guide

Open the contributing guide

Research direction

Treat this as an umbrella audit rather than a single starter task. Begin with the candidate list and inspect the named entry points in plot.py, matplotlib_backend.py, text2mat.py, and io/sources.py; use the cited examples, tutorials, and .venv checks to understand recurrence. Done means one explicitly selected candidate is implemented with appropriate regression coverage and the five launch clips remain visually identical.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-visualization
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.