v1.0.4 regression: embed-gemma:300m embeddings via --embed 1 are wrong (uncorrelated with v0.9.39 on identical weights)

Problem Description

/v1/embeddings output for embed-gemma:300m (EmbeddingGemma-300M, loaded with flm serve <chat_tag> --embed 1) changed between v0.9.39 and v1.0.4 in a way that is not a numerically-different-but-equivalent re-encoding: it is statistically uncorrelated with the correct embedding for the same input text.

Identical on-disk model weights were used for both versions (the weights cache is shared across binary versions; both binaries resolve the same installed weight files). Only the binary differs.

Expected

Cosine similarity close to 1.0 between the v0.9.39 and v1.0.4 embeddings of the same sentence, if v1.0.4 is a correct (even if numerically different) implementation of the same model.

Actual

Mean cosine similarity across 20 sentence pairs: -0.0228 (range approx. -0.10 to +0.05, n=20). For 768-dimensional unit vectors this is indistinguishable from two unrelated random vectors (expected cosine ~0, stddev ~1/sqrt(768) ~= 0.036).

Two further, independent checks show that v1.0.4's output, not v0.9.39's, is the wrong one:

• v0.9.39's embeddings for the same sentences matched embeddings recorded from an earlier v0.9.39 run of the identical model and prompts: mean cosine 0.9996.
• A from-scratch reference implementation of embed-gemma:300m (mean-token pooling -> two Dense re-projection layers -> L2 normalize, per the model's published config) disagreed with live v1.0.4 output at mean cosine -0.0378, again statistically random.

Workaround

Pin to v0.9.39 for embed-gemma:300m embedding use; v1.0.4's embedding output for this model should not be trusted until fixed.

Operating System

Microsoft Windows 11 Education, version 10.0.26200

CPU

AMD Ryzen AI MAX+ 395 w/ Radeon 8060S (Framework Desktop)

GPU

AMD Radeon 8060S Graphics (integrated). The issue is on the NPU path: "NPU Compute Accelerator Device" (XDNA), which is what FastFlowLM's --embed 1 runs on.

ROCm Version

Not applicable: Windows, FastFlowLM binary release running on the NPU; no ROCm installed.

Installation Method

Other (describe in Additional Information)

Installed ROCm Packages / Versions

None (no ROCm; FastFlowLM Windows binaries only: v1.0.4 and v0.9.39).

ROCm Component

No response

Steps to Reproduce

No response

(Optional for Linux users) Output of rocminfo --support

No response

Additional Information

Prebuilt Windows release zip from this repository's GitHub Releases: fastflowlm_1.0.4_windows_amd64.zip (bad) and fastflowlm_0.9.39_windows_amd64.zip (good), sha256-verified, extracted to separate directories.

Identical on-disk weights for both versions; only the binary differs.

Start by reproducing /v1/embeddings with embed-gemma:300m using the v0.9.39 and v1.0.4 Windows binaries, identical weights, and --embed 1 on the XDNA NPU path. Trace the embedding path for the regression and compare its output with the v0.9.39 and reference results; done means the v1.0.4 embeddings are correlated with the expected output.