Spec: UX Blocks Rework — Typage par Stages, Réduction du Catalogue et Expérience Visuelle Astryx

Problem Statement

When learners build machine learning pipelines in MLBlock, they face high cognitive load and confusing errors:

1. The Catalog presents 88 flat blocks across 13 technical categories without guidance on the machine learning lifecycle sequence.
2. Connecting mismatched data types (such as a CSV pd.DataFrame directly to a neural layer expecting torch.Tensor) causes immediate, opaque rejection (incompatible) without actionable suggestions.
3. Duplicate blocks with ephemeral weights vs persistent modules (issue #14, conv2d vs conv2d_layer) lead to silent training failures.
4. Discrepancies between frontend and backend type classification produce unexpected validation behaviors on unions and image types.

Solution

A revamped Block Catalog and Stage-based Type System that provides a guided, safe, and friction-free experience:

1. Reorganize the Catalog around 5 pedagogical Stages matching the standard ML workflow (S0 Ingest, S1 Prepare, S2 Represent, S3 Train, S4 Eval, plus isolated SX World for RL).
2. Curate the default Catalog view from 88 to ~45 essential Blocks, moving specialized activation variants into a collapsed "Advanced" section.
3. Implement a unified TypeSystem Facade synchronized across backend and frontend with explicit conversion strategies.
4. Provide interactive assistance on the ReactFlow canvas: when connecting convertible ports, present an Astryx confirmation popover offering to automatically insert the intermediary adapter Block (e.g., df_to_tensor), and display visual Stage badges on each Block.
5. Guarantee complete coverage of 12 canonical reference Pipelines sourced from official documentation (PyTorch, scikit-learn, Gymnasium).

User Stories

1. As a learner, I want to view Blocks in the palette organized by the 5 stages of the ML lifecycle (Ingest, Prepare, Represent, Train, Eval), so that I understand which Block to place next.
2. As a learner, I want the default palette to display only essential Blocks (~45), so that I am not overwhelmed by dozens of rare activation functions and duplicate blocks.
3. As a learner, I want to expand an "Advanced" palette drawer when needed, so that I can still access specialized activation functions or advanced utilities.
4. As a learner, I want to see a clear visual Stage badge (e.g. S0, S1, S2, S3, S4, SX) on each placed Block on the canvas, so that I can easily track the architectural flow of my Pipeline.
5. As a learner, when I connect two convertible ports (e.g., pd.DataFrame to torch.Tensor), I want an Astryx confirmation popover to appear at the cursor, so that I can confirm inserting the required converter Block with a single click.
6. As a learner, when I accept the conversion suggestion in the Astryx popover, I want the adapter Block (e.g., df_to_tensor) to be automatically placed between the two Blocks with the connecting edges rewired, so that my Pipeline remains valid without manual lookup.
7. As a learner, when I cancel or dismiss the Astryx conversion popover, I want the invalid connection to be aborted cleanly, so that my canvas does not enter an erroneous state.
8. As a learner, when I attempt an impossible connection (e.g., torch.optim.Optimizer into a data loader port), I want a descriptive explanation of the type mismatch rather than an unexplained red highlight, so that I understand why the connection is disallowed.
9. As a learner building an image classifier (CIFAR-10), I want to connect convolutional layers (conv2d_layer) that retain persistent weights during optimization, so that my model trains properly without weight disappearance bugs.
10. As a learner following the official PyTorch tutorials, I want reference examples (such as CIFAR-10 Blitz, Fashion-MNIST Quickstart, and MNIST from scratch) to be fully supported by the Catalog, so that I can reproduce standard educational benchmarks.
11. As a learner following scikit-learn tutorials, I want tabular Pipelines (Logistic Regression, PCA 4D→2D, and K-Means clustering on Iris) to validate cleanly without requiring artificial neural network training components, so that classic ML workflows remain simple.
12. As a learner exploring reinforcement learning, I want tabular Q-learning on CartPole to run within an isolated SX World stage, so that environment and policy ports are protected from accidental tensor pipeline connections.
13. As a developer, I want a single TypeSystem Facade governing type classifications on both frontend and backend, so that validation verdicts never disagree between the canvas and the execution engine.
14. As a developer, I want existing saved Pipelines using deprecated tensor blocks to be transparently handled via Adapters, so that existing user projects continue to load and execute.

Implementation Decisions

1. Staged Pipeline Architecture (ADR 0001):

   • The ML lifecycle is structured into 5 standard Stages: S0 Ingest (data ingestion), S1 Prepare (preprocessing / conversions), S2 Represent (S2A Deep Learning modules / S2B Classical ML models), S3 Train (S3A DL training loops / S3B ML fitting), S4 Eval (metrics and visualization), alongside SX World (RL environment/policy isolation).
   • In Stage S2A, two conceptual sub-states are recognized without adding extra stages: S2A-CNN (vision/convolutions) and S2A-Seq (NLP/embeddings/recurrent networks).

2. Unified TypeSystem Facade:

   • Single point of truth for port compatibility, family mapping (family_of), Stage association (stage_of), and conversion graph resolution.
   • Exact mirror implementation in frontend TypeScript ensuring parity on complex dtypes (including PIL images, sequence lists, and union dtypes like pd.DataFrame | numpy.ndarray).

3. Catalogue Reduction (~45 Blocks):

   • Pure stateless functional operations are merged into their canonical module counterparts (relu_layer, maxpool2d_layer, flatten_layer).
   • Deprecate standalone weightless tensor operations (conv2d, linear) in favor of *_layer counterparts (resolving issue #14).
   • Collapse advanced activation variants (gelu, selu, mish, elu, etc.) into an accessible sub-group.

4. Frontend UX via Astryx:

   • The connection drop event on a convertible edge triggers an interactive Astryx Popover dialog: "Type convertible détecté : insérer automatiquement le bloc adaptateur [Nom] ?" with [Insérer] and [Annuler] actions.
   • Astryx badge rendering on BlockNode indicating the node's Stage (S0..S4, SX) with themed color accents.
   • Palette accordion grouped by the 5 Stages, with an expandable drawer for Advanced Blocks.

5. Reference Execution Matrix:

   • 12 reference Pipelines (A1-A7 DL, B1-B3 ML, C1-C2 RL) codified in standard JSON format acting as regression acceptance suites.
   • CartPole DQN (C2) is classified as a P1 feature needing a future Adapter env ↔ tensor, keeping the initial v1 focus on C1 tabular isolation.

Testing Decisions

• Good test criteria: Tests must exercise the external interfaces of the validation and typing systems with full Pipeline payloads, avoiding coupling to internal AST traversal or catalog indexing internals.
• Backend testing: Exercised through mlblock.validation.validate(pipeline_payload). Tests will execute against all 12 reference Pipelines and assert topological validity, Stage ordering checks (stage(dst) < stage(src) rejections), and port family verdicts.
• Frontend testing: Exercised through Vitest specs mirroring test_types.py for type parity (typeCheck.test.ts), verifying conversion graph reachability, and verifying adapter node insertion logic on document state.
• Prior art: Existing unit suites in backend/mlblock/tests/test_validation.py and frontend/src/utils/typeCheck.test.ts.

Out of Scope

• Sub-pipeline nesting (Composite pattern) or multi-canvas tabs.
• Dynamic GPU execution scheduler changes (execution protocol remains untouched).
• Implementation of the P1 Adapter env ↔ tensor bridge for CartPole DQN (deferred post-v1).
• Visual canvas theme alterations beyond Astryx badge and popover integration.

Further Notes

• Tracked under roadmap docs/UX_Blocks_Rework/ and ADR docs/adr/0001-staged-typing.md.
• Directly addresses open issue #14.

Start with docs/adr/0001-staged-typing.md, backend/mlblock/tests/test_validation.py, and frontend/src/utils/typeCheck.test.ts. Run the existing backend and Vitest suites, then trace the validation and type-checking entry points before splitting the staged catalog, facade, and connection UX work. Done means the 12 reference pipelines and frontend parity tests pass, with adapter insertion and stage validation covered.