Oversized chunk increases embedding memory to 22 GiB for the whole run

@davidh233's team ran Monodex against several large monorepos and found that one very long chunk can push embedding memory to 22 GiB and hold it there for the rest of the crawl. On a 31 GiB machine the crawl cannot finish. They observed two things:

First, the chunker bounds characters, not tokens: TARGET_CHARS is 6000, but the partitioner can only cut at AST boundaries and the fallback only at line boundaries, so a minified bundle or a generated file whose body is one long line comes out as a single chunk regardless. They found one at 9,907 tokens.

Second, ONNX Runtime's arena grows to fit the longest sequence it has ever served and keeps those pages. With batch=1, attention intermediates grow roughly with the square of sequence length, so RSS is a staircase: it steps up the first time a long chunk arrives, never comes down, and multiplies by instance count since each model instance has its own arena. MAX_LENGTH in src/engine/parallel_embedder.rs is 8192, which lets those chunks reach the model at full length.

Separately, PER_INSTANCE_RAM in src/engine/system_info.rs plans 2.5 GB per instance for the "auto" heuristic, against a measured 5 to 6.5 GB. It was a padded guess and the padding went the wrong direction, so a default configuration oversubscribes memory.

Investigation

Using synthetic samples to hold everything but sequence length constant, and subtracting a 0.95 GB baseline, the plateau tracks the longest sequence seen so far: 1929 tokens gives 1.95 GB, 2749 gives 3.4 GB, 3253 gives 3.8 GB. Instrumenting the 9,907-token chunk at the 8192 cap caught one inference taking RSS from 11.8 GB to 21.9 GB, in line with the 22.2 GiB production peak. Disabling memory pattern, switching to a non-arena allocator, and MALLOC_ARENA_MAX=1 had no measurable effect. At a 4096 cap the same repository indexed end to end at a 7.07 GB peak.

They also tested 2048, and the tradeoff comes out against it: the throughput gain stayed inside run-to-run variance while the truncated share grew from 0.10% to 0.41%. The one durable result there is that a 2048 cap halves the per-instance high-water mark, which would fit indexing on a 16 GiB machine if that ever becomes a requirement.

Proposed fixes

Change MAX_LENGTH from 8192 to 4096, bump EMBEDDER_ID since the cap changes what vector a given chunk text produces, and recalibrate PER_INSTANCE_RAM against the measured high-water mark at the new cap, along with the corresponding README.md sentence. Bumping EMBEDDER_ID invalidates existing databases, which under the current pre-1.0 policy means init-db --delete-everything and a re-crawl.

Truncation costs the tail of the vector for chunks over 4096 tokens, which was 0.06% to 0.10% of chunks in their measurements. The full text is still stored, so fts still covers those chunks entirely. If the long chunks in a given repository are bundled or generated output, patternsToExclude keeps them out of the index altogether, which is usually the better answer.

Their suggested fix lowers the default to 4096 and adds a MONODEX_MAX_LENGTH environment variable to override it:

 const MODEL_ID: &str = "jinaai/jina-embeddings-v2-base-code";
-const MAX_LENGTH: usize = 8192;
+const MAX_LENGTH: usize = 4096;
 const HIDDEN_SIZE: usize = 768;

+// truncation site reads the cap through a cached env override
+fn effective_max_length() -> usize {
+    static V: OnceLock<usize> = OnceLock::new();
+    *V.get_or_init(|| std::env::var("MONODEX_MAX_LENGTH")
+        .ok().and_then(|v| v.parse().ok()).unwrap_or(MAX_LENGTH))
+}

I propose to make it a constant instead. A machine consuming a published index is not read-only. It runs incremental crawls, so vectors it produces locally land in the same table beside vectors produced elsewhere and get ranked together in one search. If the cap varies by machine, the same chunk text has two different vectors depending on which machine touched it last, and nothing records which one is which. That makes the cap part of what the embedding function means, which is why it belongs in EMBEDDER_ID rather than in configuration. Deriving it from available hardware fails the same way. modelInstances and threadsPerInstance are the right place for hardware to have an opinion, since they decide how fast vectors are produced and not what is in them.

Feedback welcome: If anyone has a repository where chunks between 4096 and 8192 tokens are real source code rather than generated output, that would change my read of the truncation cost.

@LPegasus

Start in src/engine/parallel_embedder.rs at MAX_LENGTH and in src/engine/system_info.rs at PER_INSTANCE_RAM, then check the related README.md guidance. Review the measured 4096-token memory and truncation results before resolving the constant-versus-environment-override question. Done means the selected cap, embedding identifier, instance-memory estimate, and documentation agree and the indexing tradeoff is validated.