`nemo-relay<0.8` pin makes the LangChain multimodal fix (NeMo-Relay#984) unreachable — telemetry still strips images from agent requests

Summary

NeMo-Relay fixed multimodal content loss in its LangChain codec in
NVIDIA/NeMo-Relay#984 (merged 2026-09-03,
first published in nemo-relay==0.9.0rc1 on 2026-09-11).

nemo-platform users cannot get that fix. nemo-fabric and nemo-fabric-adapters-* pin
nemo-relay>=0.7.2,<0.8 in every published release, including the newest
(0.4.0b2), and the fix was not backported to the release/0.8 or release/0.7
branches. The newest stable nemo-relay (0.8.4) is still affected.

So on nemo-platform 0.5.0 and 0.5.1, setting telemetry.enabled: true silently strips
image content blocks from the request sent to the model — no error, no warning, no log
line. An agent with a vision-capable model simply stops seeing images, and the only signal
is a drop in prompt tokens.

Impact

• Affects any LangChain/deepagents agent on nemo-platform using a vision-capable model.
• Silent. Nothing surfaces the loss; the agent answers as if the image were blank.
• Telemetry is the trigger, so observability and vision are mutually exclusive today.
• The re-encoded request is what reaches the provider, so this mutates the outbound
  payload — it is not merely a bad telemetry record.
• No workaround short of disabling telemetry: nemo_agents_plugin/.../translator.py
  rejects any telemetry.provider other than "relay", so the native/OTel path is not
  reachable from agent.yaml.

Evidence for the pin

$ python -c "import json,urllib.request as u; d=json.load(u.urlopen('https://pypi.org/pypi/nemo-fabric-adapters-deepagents/0.4.0b2/json')); print([r for r in d['info']['requires_dist'] if 'relay' in r])"
['nemo-relay<0.8,>=0.7.2; extra == "relay"', 'nemo-relay<0.8,>=0.7.2; extra == "full"']

$ python -c "import json,urllib.request as u; d=json.load(u.urlopen('https://pypi.org/pypi/nemo-fabric/0.4.0b2/json')); print([r for r in d['info']['requires_dist'] if 'relay' in r])"
['nemo-relay<0.8,>=0.7.2; extra == "relay"']

nemo-platform 0.5.0 / 0.5.1 pull nemo-fabric[claude,codex,relay]==0.3.0b1, which
carries the same pin. Latest stable nemo-relay is 0.8.4; the fix is only in 0.9.0rc1.

nemo-relay	Behaviour	Vision
0.7.3 (pulled by nemo-platform 0.5.0/0.5.1)	one annotated message per content block, scalar content	broken
0.8.4 (latest stable)	unchanged	broken
release/0.8, release/0.7	fix not backported	broken
0.9.0rc1 / main	content blocks preserved	fixed

Reproduction

Runs offline — no network, no API key, no model, no FreeCAD/MCP/platform. The
"provider" is a stub handler that records the ModelRequest it receives; anything the
stub does not receive cannot have reached a real provider either.

#!/usr/bin/env python3
"""NeMo Relay's LangChain middleware strips image content blocks from the
OUTBOUND model request. Requires no network and no credentials."""

import json

from langchain.agents.middleware import ModelRequest, ModelResponse
from langchain_core.messages import AIMessage, HumanMessage

from nemo_relay.integrations.langchain.middleware import NemoRelayMiddleware

PNG_B64 = (
    "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
    "AAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg=="
)

VISION_MESSAGE = HumanMessage(
    content=[
        {"type": "text", "text": "What is in this image?"},
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{PNG_B64}"}},
    ]
)


class _StubModel:
    model = "stub-model"


def _make_request() -> ModelRequest:
    return ModelRequest(
        model=_StubModel(), messages=[VISION_MESSAGE], system_message=None,
        tool_choice=None, tools=[], response_format=None,
        state={}, runtime=None, model_settings={},
    )


seen = {}


def handler(req: ModelRequest) -> ModelResponse:
    seen["messages"] = [{"type": m.type, "content": m.content} for m in req.messages]
    return ModelResponse(result=[AIMessage(content="ok")], structured_response=None)


handler(_make_request())
print("WITHOUT middleware:", json.dumps(seen["messages"])[:200])

NemoRelayMiddleware().wrap_model_call(_make_request(), handler)
print("WITH middleware   :", json.dumps(seen["messages"]))

Note: ModelRequest is constructed against langchain 1.4.0's nine-field dataclass. On
a different LangChain 1.x that constructor may need adjusting; the rest is
version-agnostic.

Expected

The handler receives one HumanMessage whose content is the original two-block
list, image data intact.

Actual — nemo-relay 0.7.3 and 0.8.4

WITH NemoRelayMiddleware, handler received:
[
  {"type": "human", "content": "What is in this image?"},
  {"type": "human", "content": ""}
]

baseline message count : 1
with-middleware count  : 2
image block survived   : False

The single multimodal message is split into two messages and the image block becomes
content="". Running the identical script with PYTHONPATH pointed at an unpacked
nemo-relay==0.9.0rc1 wheel passes — one message, image block intact — which confirms
the pin, not the codec, is what blocks the fix today.

Environment

• nemo-platform 0.5.0, nemo-relay 0.7.3, nemo-fabric / nemo-fabric-adapters-deepagents 0.3.0b1
• deepagents 0.7.13, langchain 1.4.0, langchain-core 1.6.2, langgraph 1.2.11
• macOS 26.5.2 arm64, Python 3.13.12

Scope

• Confined to LangChainCodec. The Relay core is not at fault: AnnotatedLLMRequest
  preserves list-valued content, and the native OpenAIChatCodec round-trips an
  image_url block unchanged.
• Text tool results are unaffected.

Ask

Any one of:

1. Backport NeMo-Relay#984 to release/0.8 and cut a 0.8.5, then widen the Fabric
   pin to >=0.8.5,<0.9. Smallest blast radius; users on the current stable line get the
   fix.
2. Widen the pin to admit 0.9.x once nemo-relay 0.9.0 is final. Cleanest, but gated
   on that release and on whatever breaking changes 0.9 carries.
3. Short term: document that telemetry.enabled: true is incompatible with
   vision-capable models on nemo-platform ≤ 0.5.1, and/or emit a warning at adapter
   startup when a request contains non-text content blocks under an affected nemo-relay
   version. Silence is the worst part of this failure mode; even a log line would make it
   diagnosable.

I have no visibility into what else 0.9 changes, so I am not recommending (1) over (2).

Start with the offline reproduction and the LangChainCodec entry point in nemo_relay.integrations.langchain.middleware; inspect nemo_agents_plugin/.../translator.py for the supported telemetry path and dependency constraint. Compare behavior on the affected relay versions and the fixed release. Done means an agreed remediation is implemented or documented, with a regression check that multimodal content is preserved or the failure is clearly surfaced.