cloudflare / cloudflare/workerd

Tail Workers implicitly activate node:diagnostics_channel publishers and add unrequested overhead

Open
#6,861 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
8.7k
Forks
739
Avg merge
2d 20h
Merged PRs (30d)
174

Description

### What happened?

When a Worker has `tail_consumers` configured, Cloudflare appears to forward all `node:diagnostics_channel` messages to the Tail Worker as `diagnosticsChannelEvents`.

This also appears to make `channel.hasSubscribers` true for application libraries that use lazy diagnostics publishing.

In our case, upgrading to GraphQL.js 17 caused production-only errors:

`Failed to publish diagnostics channel message:`

The requests still returned 200 OK, but every GraphQL request produced runtime error logs.

### Why this is surprising

We are not using OpenTelemetry, and our application does not subscribe to any diagnostics channels.

The only relevant production difference is that the deployed Worker has a Tail Worker configured.

Tail Workers are expected to observe/forward invocation output. They should not implicitly activate every library’s `diagnostics_channel` instrumentation.

### Why this matters even with clone-safe payloads

The immediate error is caused by GraphQL.js publishing payloads containing objects that cannot be structured-cloned.

But even if those payloads were clone-safe, this behavior still creates unrequested overhead:

- extra library instrumentation work
- extra object allocation
- extra Tail event volume
- extra data transport/processing in the Tail Worker
- potentially more customer-paid logging/observability cost

`tail_consumers` should not silently change application behavior from “no diagnostics emitted” to “every lazy diagnostics publisher emits.”

### Reproduction shape

1. Deploy a Worker with:
- `compatibility_flags = ["nodejs_compat"]`
- `tail_consumers` configured
- a dependency that uses `node:diagnostics_channel` and gates publishing on `channel.hasSubscribers`
2. Make a request through that dependency.
3. Observe that diagnostics publishing activates in production even though the app never subscribed.

Concrete dependency observed: `graphql@17.0.1`, which creates `graphql:*` tracing channels.

### Expected behavior

Tail Workers should not count as application-level subscribers for `node:diagnostics_channel`, or diagnostics forwarding should be explicitly configurable.

Possible fixes:

- do not make Tail Worker diagnostics forwarding affect `channel.hasSubscribers`
- add a `tail_consumers` option to disable `diagnosticsChannelEvents`
- add channel-name filtering, e.g. include/exclude `graphql:*`
- drop non-cloneable diagnostics payloads without emitting request-level errors
- document clearly that Tail Workers activate diagnostics channel publishers and may add overhead

### Workaround

We patched GraphQL.js locally so its `resolveDiagnosticsChannel()` returns `undefined`, disabling its diagnostics channel integration entirely.

That avoids the production errors, but it is a brittle workaround and disables useful instrumentation for all environments.

### Related issue

GraphQL.js issue: https://github.com/graphql/graphql-js/issues/4834

This Cloudflare issue is not only about GraphQL payload clone failures. Even if GraphQL emits clone-safe payloads, Tail Workers still appear to activate lazy diagnostics publishers and create unrequested overhead.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the behavior with nodejs_compat, tail_consumers, and graphql@17.0.1, then inspect the node:diagnostics_channel handling behind diagnosticsChannelEvents and channel.hasSubscribers. Done means Tail Workers no longer implicitly activate application publishers, or the behavior is explicitly configurable without request-level errors from non-cloneable payloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
cloud, javascript, node.js
Domain
backend, cloud, observability-sre
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.