Backend cannot connect to Azure Event Grid MQTT: the Entra token is sent as a CONNECT password instead of MQTT v5 enhanced authentication

Background, for readers who don't live in MQTT

If you're already fluent in MQTT and Event Grid, skip to Summary. Otherwise this section explains just enough to make the bug obvious.

How an MQTT client proves who it is. An MQTT client opens a session by sending a CONNECT packet. Since MQTT 3.1, that packet has carried two optional fields called Username and Password — the classic way brokers authenticate clients. Eclipse Mosquitto, the broker this repo uses locally, works exactly like that.

MQTT 5 added a second, different mechanism. MQTT 5 introduced enhanced authentication: two extra CONNECT properties named Authentication Method and Authentication Data, plus a dedicated AUTH packet for multi-step or renewed authentication. This exists precisely so that token-based schemes (OAuth, JWT) have a proper home instead of being smuggled through a field designed for passwords.

Azure Event Grid only accepts the second one. Event Grid's MQTT broker supports Microsoft Entra JWT authentication exclusively through enhanced authentication. The client must set:

• Authentication Method = OAUTH2-JWT
• Authentication Data = the raw Entra access token, with aud = https://eventgrid.azure.net/

This requires MQTT v5. Event Grid never reads the CONNECT Password field looking for a token. There is no fallback path.

Microsoft Entra JWT authentication and Azure RBAC authorization to publish or subscribe MQTT messages

Why that matters here. The backend in this repo puts the token in the Password field. Every connection to Event Grid is therefore refused, and no amount of Azure-side configuration changes that.

---

Summary

src/backend/Program.cs authenticates to the MQTT broker with WithCredentials(username, token) — the MQTT username/password path. When the broker is Azure Event Grid, the connection is always rejected, because Event Grid requires the token in the MQTT v5 enhanced-authentication fields.

The consequence is not a degraded feature. It takes down the entire backend and fails the whole Radius deployment.

Impact

MqttOrderListener is a BackgroundService. When ConnectAsync leaves the client unconnected, the next line calls SubscribeAsync, which throws MqttClientNotConnectedException. .NET's default BackgroundServiceExceptionBehavior.StopHost then stops the host, so the container exits and enters CrashLoopBackOff:

"code": "Internal",
"message": "Container state is 'Waiting' Reason: CrashLoopBackOff,
            back-off 5m0s restarting failed container=backend ..."

Because Radius waits for the container to become Ready, rad deploy fails before the frontend container is created. On an AKS environment that registers the mqtt-azure-event-grid recipe, the application is not merely missing live order streaming — it cannot be deployed at all.

To be explicit about what does not fix it: provisioning a user-assigned managed identity, federated credentials, topic spaces, permission bindings and the EventGrid TopicSpaces Publisher/Subscriber role assignments is all correct and necessary work, and the broker will still refuse every connection. The defect is in the client.

The code

https://github.com/microsoft/adaptive-apps/blob/7a200b9296ed23ff847e62ab961f18c3f500b793/src/backend/Program.cs#L198-L206

optionsBuilder = optionsBuilder
    .WithTlsOptions(tls => tls.UseTls())
    // MQTTnet v5 uses username/password credentials for broker auth.
    // Event Grid validates the JWT bearer token provided as password.
    .WithCredentials(
        string.IsNullOrWhiteSpace(azureClientId) ? "oauth2-jwt" : azureClientId,
        accessToken.Token);

_logger.LogInformation("MQTT: using OAUTH2-JWT authentication (audience: {Audience})", tokenAudience);

The comment on lines 200–201 states the incorrect assumption directly. WithAuthentication does not appear anywhere in the file — the enhanced-authentication fields are never populated. Everything else in the block is right: TLS is enabled, the audience defaults correctly, and the credential selection logic is sound.

Why this is easy to miss

The failure mode actively disguises itself:

1. AZURE_CLIENT_ID is empty in the shipped radius/app.bicep, so the code falls through to DefaultAzureCredential, which reaches IMDS and successfully obtains the node identity's token.
2. Token acquisition therefore succeeds, so the catch (CredentialUnavailableException or AuthenticationFailedException) on line 187 never fires.
3. Line 206 then logs MQTT: using OAUTH2-JWT authentication — which reads like a success message.
4. Only afterwards does Event Grid reject the CONNECT, and the visible error is MqttClientNotConnectedException from SubscribeAsync.

The logs say authentication is working, and the exception points at subscription. Nothing points at the CONNECT packet.

Reproduction

1. Deploy a Radius environment that registers recipes/mqtt-azure-event-grid:latest for Radius.Resources/mqttBrokers (this is what radius/aks-env.bicep does).
2. rad deploy radius/app.bicep against that environment.
3. trading-mqtt provisions successfully; the backend container crash-loops and the deployment fails.
4. kubectl logs deploy/backend shows MQTT: using OAUTH2-JWT authentication followed by MQTTnet.Exceptions.MqttClientNotConnectedException.

Switching the same environment to recipes/mqtt:latest (Mosquitto) makes the backend healthy immediately, because that recipe returns authMethod: 'none' and the token branch is skipped entirely. That's a workaround, not a fix — it gives up the managed broker.

Suggested fix

MQTTnet 5.1.0.1559 (already referenced in TradingBackend.csproj) supports this directly:

optionsBuilder = optionsBuilder
    .WithTlsOptions(tls => tls.UseTls())
    .WithProtocolVersion(MqttProtocolVersion.V500)
    // Event Grid reads the Entra token from the MQTT v5 enhanced-authentication
    // fields. It never inspects the CONNECT password field.
    .WithAuthentication("OAUTH2-JWT", Encoding.UTF8.GetBytes(accessToken.Token));

Two details worth getting right in the same change:

• Client ID must be the Entra object ID. For Entra-authenticated clients, Event Grid expects the CONNECT client identifier to match the authenticating principal's object ID. The current $"backend-{Guid.NewGuid():N}" on line 148 will not satisfy that. (This is likely the same underlying area as #35, though that issue reports a different symptom.)
• Tokens expire. Event Grid disconnects a client when its token expires unless the client re-authenticates by sending an AUTH packet with reason code 25 and a fresh token. Without this, the backend will drop off the broker roughly hourly. Worth handling in the same pass.

Related robustness problems

These are separate from the main bug but were exposed by it, and each is cheap to fix:

1. ConnectAsync is unguarded. Line 234 is not wrapped in a try, and the only catch in the method covers token acquisition. Any broker unavailability — a restart, a transient network fault — therefore terminates the whole backend process rather than retrying. Consider either catching and retrying with backoff, or setting BackgroundServiceExceptionBehavior.Ignore if a degraded backend is preferable to a dead one. A REST API that serves accounts, orders and trades probably shouldn't be killed by a message broker being briefly unreachable.

2. radius/app.bicep reads the MQTT auth method from the wrong resource. It wires MQTT_AUTH_METHOD and MQTT_TOKEN_AUDIENCE from the workload identity:

   MQTT_AUTH_METHOD: { value: backendIdentity.properties.authMethod }
   

   and radius/recipes/workload-identity/azure-workload-identity.bicep hard-codes authMethod: 'OAUTH2-JWT'. So on any Azure environment the application attempts token authentication regardless of which mqttBrokers recipe the environment registered — the identity silently overrides the platform operator's choice of broker. Registering the Mosquitto recipe on AKS does not work until this is corrected.

   The mqttBrokers resource type already exposes both values, and radius/resource-types/types.yaml documents them as the intended source (mqtt.properties.authMethod, mqtt.properties.tokenAudience). Reading them from the broker is a one-line change per container, is identical in every environment, and lets each recipe answer for itself:

   MQTT_AUTH_METHOD:    { value: tradingMqtt.properties.authMethod }
   MQTT_TOKEN_AUDIENCE: { value: tradingMqtt.properties.tokenAudience }
   

3. AZURE_TENANT_ID is wired to an empty parameter. app.bicep passes the workloadIdentityTenantId parameter straight through, and it defaults to '', so containers receive an empty tenant even though the workload-identity recipe returns the correct one via tenantId. This also prevents the WorkloadIdentityCredential branch on line 153 from ever being selected, since it requires a non-empty tenant. Deriving it from the identity resource (with the parameter as an override) fixes both.

4. The frontend has no Event Grid path at all. src/frontend/public/index.html connects with the browser mqtt.js client to MQTT_WS_URL with no credentials. Against Event Grid that cannot work — it needs a token and a browser-reachable endpoint. This may be intentional scope, but it's worth stating in the docs so operators know that selecting the Event Grid recipe does not give them browser-side streaming either.

5. The Event Grid recipe returns resources: []. radius/recipes/mqtt/azure-event-grid.bicep deliberately reports no resources (the comment cites Radius Azure scope validation). A side effect is that Radius does not track the namespace it created, so switching recipes or running rad app delete orphans the Event Grid namespace and it keeps billing. A note in the recipe header would save people money.

Environment

• Repo at 7a200b9296ed23ff847e62ab961f18c3f500b793
• Radius on AKS (Kubernetes 1.31), Radius.Resources/* types from radius/resource-types/types.yaml
• mqttBrokers bound to ghcr.io/microsoft/adaptive-apps/recipes/mqtt-azure-event-grid:latest
• Event Grid namespace with MQTT enabled, reached over TLS on 8883
• MQTTnet 5.1.0.1559, Azure.Identity 1.13.2, .NET 8

Happy to open a PR for the WithAuthentication change and the app.bicep wiring if that's useful — I've already validated the recipe-selection and wiring fixes end to end on AKS.

Note on #35

This is distinct from #35, which reports trading-mqtt failing with an empty subscription identifier during provisioning. Here the resource provisions cleanly and the failure is at the application's connect time. The two may share a root cause around Entra client identity, but they reproduce independently.

Start in src/backend/Program.cs around the MQTT client setup and read TradingBackend.csproj to confirm the MQTTnet version and supported authentication APIs. Then inspect radius/app.bicep, radius/resource-types/types.yaml, and radius/recipes/mqtt/azure-event-grid.bicep for broker-property wiring. Done means Event Grid accepts the backend connection and token lifecycle, broker selection supplies the intended settings, and the documented deployment path no longer crash-loops.