[Hermes] Langfuse plugin because it includes the unresolved public key in protobuf payload

### Investigation Summary

- Hermes traces cannot be successfully sent to a Langfuser server because OpenShell blocks it
- This happens when using a new Langfuse OpenShell provider which includes both public and private key
- The two credentials are succesfully resolved by OpenShell for authentication
- .. but the Langfuse SDK also sends the public key in the payload
- .. and OpenShell/NemoClaw do NOT resolve the credential
- .. when this reaches the OpenShell boundary
- .. something black magic OpenShell happens
- .. and the request is blocked

### Description

Long story short:

The Languse SDK sends the unresolved public key as part of the (protobuf?) payload. OpenShell intercepts (?) it and blocks it.
I validated that this is the actual issue in 2 ways:

- I provided the actual public key (by adding it to `~/hermes/.env`) inside the sandbox and it works (however I cannot then restart the gateway or access the sandbox because nemoclaq complains about the plain credential inside `~/.hermes/.env`).
- I bake a modified Langfuse plugin (modified a couple of python files) into the base image. The modifications still use the OpenShell credentials for authentication purposes but then swap in the unresolved public key with the actual public key just before sending the payload to the Langfuse server.

Courtesy of Codex:

# Root-cause analysis: Langfuse OTLP export fails with OpenShell credentials

## Executive summary

The Langfuse Python SDK receives an OpenShell resolver placeholder as its public_key. It correctly uses that value in authentication headers, but it also serializes the public key
into the OTLP protobuf request body as an instrumentation-scope attribute.

OpenShell can replace the credential in the configured HTTP authentication fields, but this provider has no protobuf request-body rewrite. When OpenShell finds the unresolved
placeholder still present inside the request body, it fails closed and returns HTTP 500 rather than allowing the placeholder to leave the sandbox.

The concise root cause is:

> NemoClaw passes an OpenShell credential placeholder into a Langfuse SDK field that is not header-only. The SDK copies that value into the OTLP protobuf body, where OpenShell
> cannot rewrite it and therefore rejects the request.

The primary integration defect is in the NemoClaw/Hermes Langfuse compatibility layer. OpenShell is enforcing its credential boundary correctly, although its log incorrectly says
the unresolved placeholder was found “in header.”

## Expected credential flow

NemoClaw’s intended security model is:

1. The real credential remains in the host-side OpenShell gateway.
2. The sandbox sees only a resolver placeholder such as:

openshell:resolve:env:v11178089249861885626_LANGFUSE_SECRET_KEY

3. The application sends the placeholder through an approved request field.
4. The OpenShell L7 proxy replaces that field with the real credential at egress.
5. The upstream service receives the real credential, while the sandbox never does.

This matches the documented model: OpenShell stores provider credentials and substitutes placeholders at approved egress boundaries. See Credential Storage and How NemoClaw Works.

That model only works when every occurrence of the placeholder appears in a request location OpenShell knows how to rewrite.

## What actually happens

The failing request follows this path:

NemoClaw/OpenShell provider
│
│ exposes versioned placeholders
▼
Hermes environment
HERMES_LANGFUSE_PUBLIC_KEY=openshell:resolve:env:v..._LANGFUSE_PUBLIC_KEY
HERMES_LANGFUSE_SECRET_KEY=openshell:resolve:env:v..._LANGFUSE_SECRET_KEY
│
▼
Hermes Langfuse plugin
constructs Langfuse(public_key=..., secret_key=...)
│
▼
Langfuse Python SDK
├─ creates Authorization: Basic ...
├─ creates x-langfuse-public-key: ...
└─ embeds public_key into OTLP protobuf instrumentation metadata
│
▼
OpenShell L7 proxy
├─ matches _provider_langfuse_hermes
├─ allows the HTTP request
├─ can handle the configured authentication fields
└─ finds an unresolved placeholder in the protobuf body
│
▼
HTTP 500 credential_injection_failed
│
└─ request does not reach the Langfuse server

The critical detail is that the public key is not treated by the Langfuse SDK as an authentication-header-only value.

The SDK debug output proves that it becomes telemetry data:

instrumentationScope.attributes.public_key =
openshell:resolve:env:v..._LANGFUSE_PUBLIC_KEY

The OpenTelemetry exporter then serializes that attribute into the protobuf request body.

## Component responsibilities

### NemoClaw

NemoClaw is responsible for:

- Configuring the Hermes sandbox.
- Registering provider credentials with OpenShell.
- Placing resolver placeholders into sandbox-visible configuration.
- Patching Hermes so that its Langfuse validator accepts those placeholders.
- Ensuring that placeholders are used only in locations OpenShell can safely rewrite.

The current patch assumes that the Langfuse SDK will turn the placeholders into authentication headers:

> “before the Langfuse SDK can turn them into outbound authentication headers”

That assumption is incomplete. The relevant comment is in agents/hermes/patch-langfuse-credentials.mts:12.

The SDK does produce authentication headers, but it also puts the public key into the OTLP body. The patch therefore makes Hermes accept a configuration that cannot complete the
OpenShell credential-injection flow.

For this integration, NemoClaw is the primary owner of the fix.

This does not mean NemoClaw’s general credential architecture is wrong. The defect is specifically the assumption that both Langfuse keys can safely be represented as resolver
placeholders when passed directly into the Langfuse SDK.

### Hermes and its Langfuse plugin

Hermes is responsible for:

- Reading HERMES_LANGFUSE_PUBLIC_KEY and HERMES_LANGFUSE_SECRET_KEY.
- Falling back to the canonical LANGFUSE_* variables where applicable.
- Validating the values.
- Passing them to the Langfuse SDK.
- Creating and managing the Langfuse client used by Hermes.

Hermes is not responsible for resolving OpenShell placeholders. It does not know the underlying secrets and should not know them.

The original Hermes validator rejecting values that do not start with pk-lf- or sk-lf- actually prevented this broken configuration from progressing. NemoClaw’s patch relaxed that
validator.

Hermes itself is not the root cause because the same failure was reproduced using the Langfuse SDK directly from Python inside the sandbox, without Hermes:

Langfuse(
public_key=public_key,
secret_key=secret_key,
base_url=base_url,
)

That reproduction removes Hermes runtime behavior from the failing path.

A Hermes change could make integration easier—for example, supporting separate non-secret public-key configuration—but Hermes cannot solve OpenShell credential rewriting on its
own.

### Langfuse Python SDK

The Langfuse SDK is responsible for:

- Accepting the public key, secret key, and base URL.
- Creating the authentication headers.
- Decorating OpenTelemetry data with Langfuse metadata.
- Sending OTLP protobuf requests through the OpenTelemetry exporter.

The SDK treats the supplied public key as an ordinary string. It has no knowledge of the openshell:resolve:env: syntax and is not expected to resolve it.

Its important behavior here is:

- The public key is used in request headers.
- The public key is also added to OTLP instrumentation metadata.
- That metadata becomes part of the serialized protobuf body.

The SDK is therefore the component that introduces the placeholder into the body, but this is not necessarily a Langfuse SDK bug. A Langfuse public key is an identifier, and
including that identifier in telemetry metadata can be valid SDK behavior.

It becomes a problem only because the integration supplied an unresolved credential sentinel instead of an actual public-key value.

The SDK could theoretically offer an option not to serialize the public key into instrumentation metadata. That would be a possible compatibility feature, but NemoClaw should not
assume such behavior without a supported SDK contract and a regression test.

### OpenTelemetry HTTP exporter

The OpenTelemetry exporter is responsible for:

- Serializing spans into OTLP protobuf.
- Sending the HTTP POST request.
- Retrying transient HTTP 500 responses.
- Reporting export success or failure.

It is not responsible for credentials or OpenShell placeholders.

The warnings such as:

Transient error Internal Server Error encountered while exporting span batch

are downstream symptoms. The exporter receives a 500 from the local forward proxy and retries it according to its normal retry policy.

The final Langfuse message:

Successfully flushed OTEL tracer provider

only means that the flush operation completed. It does not mean the spans were successfully accepted by Langfuse. The preceding exporter error is the meaningful result.

### OpenShell

OpenShell is responsible for:

- Keeping the real credentials outside the sandbox.
- Attaching the provider to the sandbox.
- Enforcing network policy.
- Matching the request to the appropriate provider policy.
- Replacing credential placeholders at approved egress locations.
- Refusing to forward unresolved credential placeholders.

The following line proves that network policy and provider matching succeed:

ALLOWED ... POST http://172.17.0.1:3000/api/public/otel/v1/traces
[policy:_provider_langfuse_hermes engine:l7]

ALLOWED means the request was permitted by network policy. It does not guarantee that credential injection later succeeded.

The subsequent warning identifies the actual failure:

credential injection failed ...
credential placeholder could not be resolved in header

OpenShell’s rejection is appropriate from a security perspective. Forwarding a request that still contains openshell:resolve:env:... would leak an internal credential sentinel and
would send a request with invalid authentication or telemetry data.

However, OpenShell has a diagnostic defect: the unresolved placeholder was demonstrated to be in the request body, but the log says “in header.” That message is inaccurate or
overly generic. OpenShell should report the actual location, for example:

unresolved credential placeholder remained in request body

So OpenShell owns a secondary logging/diagnostic issue, not the primary integration defect.

### The OpenShell Langfuse provider profile

The provider profile is responsible for:

- Declaring credential keys.
- Defining which traffic belongs to the provider.
- Defining how credentials are inserted or rewritten.
- Contributing provider-specific network policy.

The provider is attached and its policy matches the request:

langfuse-hermes langfuse-hermes-v1 2 0

The profile reports:

request_body_credential_rewrite: null

That does not mean OpenShell should ignore placeholders in request bodies. It means the profile does not define a body-rewrite operation.

Consequently:

- Header-based injection can work.
- Arbitrary protobuf-body replacement is unavailable.
- A residual placeholder in the body is rejected.

Adding a generic string replacement to a binary protobuf body would be unsafe and brittle. Protobuf fields have encoded lengths, the body may be compressed, and blind mutation can
corrupt the payload. A body rewrite would need to understand the OTLP protobuf schema or operate before serialization.

The profile configuration is therefore consistent with the observed behavior. It is not merely missing an obvious header field.

### Langfuse server

The Langfuse server is responsible for:

- Authenticating requests that reach it.
- Parsing the OTLP protobuf body.
- Ingesting valid telemetry.

It is not responsible for this failure because the failing SDK request is rejected by OpenShell before reaching Langfuse.

The control tests prove that Langfuse is reachable:

- Correctly authenticated control request: HTTP 200.
- Ordinary malformed protobuf body: HTTP 400 with:

{"error":"Failed to parse OTel Protobuf Trace"}

That 400 is a Langfuse response, proving that the request crossed the proxy and reached the server.

By contrast, a body containing the placeholder returns:

{
"detail": "unresolved credential placeholder in request",
"error": "credential_injection_failed"
}

That is an OpenShell response, proving that the request stopped at the proxy.

### Dockerfile and sandbox rebuild

The Dockerfile and its SHA-256 pin are responsible only for ensuring that the intended patch is copied into the image without modification.

They are not responsible for the runtime failure. The debug logging proved that the modified patch was installed and executed correctly.

Likewise, --fresh, --recreate-sandbox, and provider attachment are not the cause. Rebuilding the same configuration reproduces the same incompatibility.

## Why the control requests pass

The successful curl and requests.post() controls show that all of the following work:

- The base URL is correct.
- The Langfuse service is reachable from the sandbox.
- The OpenShell network policy allows the request.
- The langfuse-hermes provider policy matches.
- Versioned credential placeholders can be used in the expected HTTP authentication path.
- Basic authentication has the correct shape.
- x-langfuse-public-key is not independently causing the failure.
- Python requests and the sandbox proxy environment work.

The meaningful difference between the successful control request and the failing SDK request is the body.

The direct body experiment proves that causality:

Request Result Meaning
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Empty authenticated control 200 Header credential path works
────────────────────────────────────── ──────── ────────────────────────────────────────────
Ordinary invalid protobuf body 400 Request reaches Langfuse
────────────────────────────────────── ──────── ────────────────────────────────────────────
Body containing resolver placeholder 500 OpenShell rejects the residual placeholder
────────────────────────────────────── ──────── ────────────────────────────────────────────
Langfuse SDK OTLP body 500 SDK body contains that placeholder

## Why raw credentials work

Saving the actual Langfuse keys in ~/.hermes/.env works because the SDK then serializes the actual public key into the protobuf body.

There is no unresolved openshell:resolve:env: sentinel for OpenShell to reject.

This is useful diagnostic evidence, but placing the secret key directly inside the sandbox defeats the intended OpenShell credential boundary and should not be treated as the
production fix.

## The likely correct integration model

The smallest viable design is probably:

- Treat the Langfuse public key as non-secret configuration, subject to an explicit product/security decision.
- Keep the Langfuse secret key in the OpenShell credential store.
- Give the SDK the actual public key.
- Give it the versioned resolver placeholder for the secret key.
- Let OpenShell inject the real secret into the authentication request at egress.

Conceptually:

HERMES_LANGFUSE_PUBLIC_KEY=
HERMES_LANGFUSE_SECRET_KEY=openshell:resolve:env:v..._LANGFUSE_SECRET_KEY
HERMES_LANGFUSE_BASE_URL=http://172.17.0.1:3000

This still needs an end-to-end test. In particular, the mixed actual-public-key/placeholder-secret request must be verified before declaring the fix complete.

The public key may be non-secret in Langfuse’s model, but it can still identify a tenant or project. NemoClaw should make an explicit decision about whether exposing it inside the
sandbox is acceptable.

A stronger architectural alternative is a host-side OTLP relay:

1. The sandbox emits credential-free OTLP to a fixed local collector.
2. The host-side collector holds the Langfuse credentials.
3. The collector forwards telemetry to Langfuse.

That keeps both keys outside the sandbox and avoids putting OpenShell placeholders into protobuf telemetry. It is more work and creates a new managed component, so NemoClaw’s
product scope gate requires an accepted design decision before making it canonical.

## Secondary defects discovered

### Unversioned placeholders are incorrectly accepted by the patch

The current patch accepts both:

openshell:resolve:env:LANGFUSE_PUBLIC_KEY

and:

openshell:resolve:env:v..._LANGFUSE_PUBLIC_KEY

Your tests showed that the unversioned form fails with the live gateway.

Therefore, the patch’s validation and tests currently accept a value that the actual OpenShell integration cannot resolve. That should be corrected independently of the protobuf-
body issue.

### The patch does not enforce canonical equality

The debug code calculates:

canonical_equal=true|false

but the acceptance condition only checks the placeholder’s textual shape:

if raw_prefix_match or placeholder_shape_match:
return None

Thus, a stale but syntactically valid versioned placeholder can pass validation even when it differs from the canonical sandbox environment value.

If the patch remains, placeholder acceptance should require equality with the canonical value, not merely a matching regular expression.

### OpenShell reports the wrong request location

The proxy reports:

credential placeholder could not be resolved in header

even when a controlled test places the only unresolved placeholder in the body.

This is an OpenShell observability bug. It materially delayed diagnosis because it directed investigation toward headers that were already working.

## Final ownership assessment

- Primary fix owner: NemoClaw’s Hermes/Langfuse integration.
- Compatibility trigger: Langfuse Python SDK serializes public_key into OTLP metadata.
- Not the root cause: Hermes runtime, because direct SDK use reproduces the problem.
- Correct security enforcement: OpenShell refuses a residual credential placeholder in the body.
- OpenShell defect: It incorrectly describes the body failure as a header failure.
- Not responsible: Langfuse server, because the failed requests never reach it.
- Not responsible: Dockerfile SHA, onboarding flags, base URL, Python proxy settings, or provider attachment.
- Unsafe workaround: Storing the real secret key in ~/.hermes/.env.
- Candidate proper fix: Actual public key as approved non-secret config plus an OpenShell-managed secret key, or a host-side OTLP relay.

A good issue title would be:

> Hermes Langfuse integration embeds the OpenShell public-key placeholder in the OTLP protobuf body, causing fail-closed credential injection errors

And the one-sentence acceptance criterion should be:

> A real Langfuse SDK span exported from a Hermes sandbox reaches the configured Langfuse endpoint without placing a secret in sandbox-visible state and without leaving any
> openshell:resolve:env: marker in the serialized OTLP request body.

### Reproduction Steps

1. Create a Langfuse provider and import it:

```yaml
id: langfuse-hermes-local-v1
display_name: Langfuse for Hermes
description: Langfuse tracing from the managed Hermes Python runtime
category: data

credentials:
- name: public_key
description: Langfuse project public key
env_vars:
- LANGFUSE_PUBLIC_KEY
required: true
auth_style: basic
- name: secret_key
description: Langfuse project secret key
env_vars:
- LANGFUSE_SECRET_KEY
required: true
auth_style: basic

endpoints:
- host: ${LANGFUSE_BASE_URL}
port: ${LANGFUSE_PORT}
protocol: rest
enforcement: enforce
allowed_ips:
- 172.18.0.1/32
rules:
- allow: { method: GET, path: "/api/public/**" }
- allow: { method: POST, path: "/api/public/**" }

binaries:
- /opt/hermes/.venv/bin/python3

inference_capable: false
```

2. Add the credentials:

```shell
nemoclaw credentials add langfuse-hermes \
--type langfuse-hermes-local-v1 \
--credential LANGFUSE_PUBLIC_KEY \
--credential LANGFUSE_SECRET_KEY
```

3. Onboard/create the sandbox
4. Enable the Langfuse plugin from Hermes
5. Start Hermes and use it
6. The `~/.hermes/logs/agent.log` shows that the Langfuse plugin is loaded and that it started a tracing request
7. The same log will show that the connection to the server failed.

Workaround

1. Create a different Langfuse provider and import it:

```yaml
id: langfuse-hermes-local-v1
display_name: Langfuse for Hermes
description: Langfuse tracing from the managed Hermes Python runtime
category: data

credentials:
- name: secret_key
description: Langfuse project secret key
env_vars:
- LANGFUSE_SECRET_KEY
required: true
auth_style: basic

endpoints:
- host: ${LANGFUSE_BASE_URL}
port: ${LANGFUSE_PORT}
protocol: rest
enforcement: enforce
allowed_ips:
- 172.18.0.1/32
rules:
- allow: { method: GET, path: "/api/public/**" }
- allow: { method: POST, path: "/api/public/**" }

binaries:
- /opt/hermes/.venv/bin/python3

inference_capable: false
```

2. Add the credentials:

```shell
nemoclaw credentials add langfuse-hermes \
--type langfuse-hermes-local-v1 \
--credential LANGFUSE_SECRET_KEY
```

3. Onboard/create the sandbox
4. Add the plain credential publick key to `~/.hermes/.env`
5. Enable the Langfuse plugin from Hermes
6. Start Hermes and use it
7. The `~/.hermes/logs/agent.log` shows that the Langfuse plugin is loaded and that it started a tracing request
8. The Langfuse server receives the trace

### Environment

- NemoClaw 0.0.119
- Ubuntu 22.04.5 LTS
- Docker version 29.5.3, build d1c06ef
- Node is managed by NemoClaw

### Debug Output

[nemoclaw-debug.tar.gz](https://github.com/user-attachments/files/31831687/nemoclaw-debug.tar.gz)

### Logs

```shell

```

### Checklist

- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate

Start with agents/hermes/patch-langfuse-credentials.mts:12 and reproduce the direct Langfuse Python SDK path described in the report. Trace how the public-key placeholder reaches OTLP instrumentation metadata and determine a supported compatibility boundary that keeps it out of the protobuf body. Done means Hermes traces reach Langfuse without unresolved placeholders leaving the sandbox, with the credential flow still protected.

Written by the indexing model from the issue text.