Show & tell: non-autoregressive knowledge-graph decoding from SONAR embeddings (+ empirical findings about the embedding space)

Hi SONAR team — sharing downstream work built entirely on your model, plus some empirical findings about the embedding space that may interest you.

## What I built

**Cogito Estella**: small non-autoregressive decoder heads (5–45M params) that map a single SONAR sentence embedding directly to a knowledge graph (entities + typed relations) in one forward pass — no text decoding anywhere. The project started after reading the LCM paper; the question was whether the concept space could support *structured* decoding instead of autoregressive text.

It can. Held-out triple-F1, splits by combination (no duplicate leakage):

| Modality | F1 |
|---|---|
| Tool-call structures | **1.000** |
| Python code (AST oracle) | **0.781** (with LoRA-adapted encoder) |
| Prose, entity-conditioned | **0.827** (validated on never-touched slices) |
| Prose, open-vocab | 0.651 |

Decoder runs at 0.013 ms/sentence on a consumer RTX 5070; everything (training included) was done on that single 12GB card. Cross-lingual extraction works zero-shot: Spanish input against an English-only entity vocabulary lands on the correct nodes.


- Repo (Apache-2.0, reproducible recipes, 113 tests): https://github.com/DeliVali/cogito-estella
- Weights: https://huggingface.co/DeliVali/cogito-estella

## Empirical findings about SONAR that might interest you

1. **Role/direction information survives better than linear probes suggest.** Linear and MLP probes for agent/patient direction score near chance on the raw embedding, yet after training our graph decoder, only **0.3%** of its errors are direction reversals — the information is there, non-linearly entangled.
2. **Exact digits are recoverable at the data level.** A single-label probe fails on unseen integers (~0 exact), but digits-as-nodes plus spacing the digits in the source text ("400" → "4 0 0") reaches **98.3%** exact on unseen values. The embedding encodes per-digit information.
3. **Entity identity is the binding constraint for open-vocab decoding.** The model's own top-16 proposals contain the correct entity 90% of the time, but no second stage we tried could exploit that beyond direct argmax — suggesting a discriminative-information limit of the single-vector representation for 20k-way entity selection.
4. **Label consistency dominates label density when distilling into the space.** LLM-generated training labels (denser, semantically richer) were unlearnable (0.26 F1) versus a dumb-but-deterministic dependency-parse oracle (0.73+): inconsistent labels break distillation over sentence embeddings.

All experiments, including the failed ones, are documented in the changelog.

## Questions

- Is per-slot/structured decoding from SONAR space something the team explored internally for the LCM line? Any pointers to related internal findings would be gold.
- The encoder weights being CC-BY-NC is the main blocker for commercial

No SONAR file, test, or entry point is named; start by reviewing the linked Cogito Estella repository, its reproducible recipes, 113 tests, and changelog, then compare the claims with SONAR's existing LCM materials. Done means providing relevant public pointers or a maintainer response, including clarification of the encoder licensing question.

Written by the indexing model from the issue text.