cloudflare / cloudflare/workerd
Tail Workers implicitly activate node:diagnostics_channel publishers and add unrequested overhead
- Dominant language
- C++
- Stars
- 8.7k
- Forks
- 739
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 174
Description
### What happened?
When a Worker has `tail_consumers` configured, Cloudflare appears to forward all `node:diagnostics_channel` messages to the Tail Worker as `diagnosticsChannelEvents`.
This also appears to make `channel.hasSubscribers` true for application libraries that use lazy diagnostics publishing.
In our case, upgrading to GraphQL.js 17 caused production-only errors:
`Failed to publish diagnostics channel message:`
The requests still returned 200 OK, but every GraphQL request produced runtime error logs.
### Why this is surprising
We are not using OpenTelemetry, and our application does not subscribe to any diagnostics channels.
The only relevant production difference is that the deployed Worker has a Tail Worker configured.
Tail Workers are expected to observe/forward invocation output. They should not implicitly activate every library’s `diagnostics_channel` instrumentation.
### Why this matters even with clone-safe payloads
The immediate error is caused by GraphQL.js publishing payloads containing objects that cannot be structured-cloned.
But even if those payloads were clone-safe, this behavior still creates unrequested overhead:
- extra library instrumentation work
- extra object allocation
- extra Tail event volume
- extra data transport/processing in the Tail Worker
- potentially more customer-paid logging/observability cost
`tail_consumers` should not silently change application behavior from “no diagnostics emitted” to “every lazy diagnostics publisher emits.”
### Reproduction shape
1. Deploy a Worker with:
- `compatibility_flags = ["nodejs_compat"]`
- `tail_consumers` configured
- a dependency that uses `node:diagnostics_channel` and gates publishing on `channel.hasSubscribers`
2. Make a request through that dependency.
3. Observe that diagnostics publishing activates in production even though the app never subscribed.
Concrete dependency observed: `graphql@17.0.1`, which creates `graphql:*` tracing channels.
### Expected behavior
Tail Workers should not count as application-level subscribers for `node:diagnostics_channel`, or diagnostics forwarding should be explicitly configurable.
Possible fixes:
- do not make Tail Worker diagnostics forwarding affect `channel.hasSubscribers`
- add a `tail_consumers` option to disable `diagnosticsChannelEvents`
- add channel-name filtering, e.g. include/exclude `graphql:*`
- drop non-cloneable diagnostics payloads without emitting request-level errors
- document clearly that Tail Workers activate diagnostics channel publishers and may add overhead
### Workaround
We patched GraphQL.js locally so its `resolveDiagnosticsChannel()` returns `undefined`, disabling its diagnostics channel integration entirely.
That avoids the production errors, but it is a brittle workaround and disables useful instrumentation for all environments.
### Related issue
GraphQL.js issue: https://github.com/graphql/graphql-js/issues/4834
This Cloudflare issue is not only about GraphQL payload clone failures. Even if GraphQL emits clone-safe payloads, Tail Workers still appear to activate lazy diagnostics publishers and create unrequested overhead.
Contributor guide
Research direction
Start by reproducing the behavior with nodejs_compat, tail_consumers, and graphql@17.0.1, then inspect the node:diagnostics_channel handling behind diagnosticsChannelEvents and channel.hasSubscribers. Done means Tail Workers no longer implicitly activate application publishers, or the behavior is explicitly configurable without request-level errors from non-cloneable payloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cloud, javascript, node.js
- Domain
- backend, cloud, observability-sre
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100