software-mansion / software-mansion/react-native-executorch

[RNE Rewrite] Add LFM2.5 ColBERT (late-interaction multi-vector) embedding pipeline

Open
#1,306 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

improvement refactoring
Dominant language
TypeScript
Stars
1.7k
Forks
96
Avg merge
20h 51m
Merged PRs (30d)
59

Description

Summary

Add a LFM2.5-ColBERT-350M late-interaction (multi-vector) embedding pipeline under the new flow. Follow-up to the single-vector embeddings work in #1292 / #1247.

Why it isn't covered by the text embedder

ColBERT is a late-interaction model: forward returns a per-token matrix [numTokens, 128] (Linear 1024→128 + per-token L2-norm), not a single pooled vector. Relevance is scored with MaxSim (for each query token, the max cosine against all document tokens, summed) rather than a single cosine. So createTextEmbedder / useTextEmbedder cannot represent it — it needs its own pipeline.

The export scripts already exist (exporter_models/lfm2_5_colbert_350m, xnnpack + mlx) and an HF repo (react-native-executorch-lfm2.5-colbert-350m, currently only v0.9.0 / old flow).

Scope

Export (export-scripts, lfm2_5_colbert_350m)

  • Port both backends to the new flow (dynamic seq + get_dynamic_dims_forward companion), matching the embedding export
  • MLX must be built with the software-mansion-labs/executorch fork + MLX-native quantization — a pip/torchao-built MLX delegate fails at execute with Error::InvalidProgram (same root cause as the #1292 MLX fix; the MLX backend is version-locked to the runtime MLXBackend.xcframework)
  • Publish xnnpack + mlx to HF react-native-executorch-lfm2.5-colbert-350m v0.10.0

Library

  • New task createColbertEmbedder + useColbertEmbedder hook returning per-token embeddings [numTokens, 128]
  • MaxSim late-interaction scoring helper (query vs. document token matrices)
  • models.ts entries (xnnpack 8da4w + mlx int4) + index exports; download-libs already enables the MLX backend for textEmbeddings

Demo (nlp)

  • Late-interaction retrieval screen (or extend Text Embeddings) using MaxSim; MLX chip disabled on Android (MLX is iOS-only)

Notes

  • The exporter NOTES document that XNNPACK lowering bakes the dynamic seq dim to static [1,512] / [1,512,128] on the current ET build; decide between static + runtime padding vs. a build that preserves the dynamic seq (the embedding MLX variant shipped dynamic; both static+pad and dynamic were verified on device).
  • Related: standalone on-device vector store #1303.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the single-vector embedding work in #1292/#1247 and the exporter_models/lfm2_5_colbert_350m scripts first. Then trace models.ts, index exports, download-libs, and the nlp demo to understand the library and UI integration points. Done means both backends are published as v0.10.0, the ColBERT APIs and MaxSim helper work, and the demo handles MLX's iOS-only support.

Written by the indexing model from the issue text.

Assessment

Tech stack
pytorch, react-native, typescript
Domain
machine-learning, mobile
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.