[Bug] NativeConnection.withAbortSignal never cancels requests, so Activity cancellation cannot abort getClient() calls

What are you really trying to do?

Running TypeScript Workers in Kubernetes. Activities call back into Temporal via getClient(). When a Worker shuts down — gracefully or after a fatal error — we expect Activity cancellation to abort those in-flight calls so the Worker can finish draining and the process can exit, letting the orchestrator replace the pod.

Describe the bug

NativeConnection.withAbortSignal documents that it cancels in-flight requests, but never does. Its JSDoc (packages/worker/src/connection.ts, worker/lib/connection.js:150-169) states:

Set an AbortSignal that, when aborted, cancels any ongoing service requests executed in fn's scope. This will locally result in the request call throwing a ServiceError with code CANCELLED

The implementation only stores the signal in AsyncLocalStorage (worker/lib/connection.js:170-172):

async withAbortSignal(abortSignal, fn) {
  const cc = this.callContextStorage.getStore();
  return await this.callContextStorage.run({ ...cc, abortSignal }, fn);
}

and nothing consumes it. sendRequest reads only metadata and deadline, behind an explicit TODO (worker/lib/connection.js:91):

// TODO: add support for abortSignal
const ctx = this.callContextStorage.getStore() ?? {};
const metadata = ctx.metadata != null ? tagMetadata(ctx.metadata) : {};
const req = {
  rpc: method.name, req: requestData, retry: true, metadata,
  timeout: ctx.deadline ? getRelativeTimeout(ctx.deadline) : null,
};

The gRPC Connection does implement it — abortSignal.addEventListener('abort', () => call.cancel()) (client/lib/connection.js:318-319) — so the two connection types differ silently despite sharing the documented contract.

Why this bites Activities. The Worker wraps every Activity in this._client.withAbortSignal(this.abortController.signal, …) (worker/lib/activity.js:149) and aborts with CancelledFailure(WORKER_SHUTDOWN) on shutdown (activity.js:49-55). That client is the Activity context's client — getClient() returns Context.current().client (activity/lib/index.js:423-425), which the Worker builds over its own NativeConnection (worker/lib/worker.js:538-550), and NativeConnection routes every workflowService RPC through the sendRequest above (worker/lib/connection.js:65).

So the cancellation machinery is wired end to end and correct — the abort genuinely fires — but it cannot cancel anything an Activity does through getClient(). Combined with timeout: null when no deadline is set, such a call is both uncancellable and unbounded.

Consequence: an Activity blocked on a getClient() call cannot be cancelled at shutdown, so the graceful drain never completes and worker.run() never settles — it neither resolves nor rejects, and nothing is logged after Initiating Worker shutdown.

This is not specific to fatal errors. We observe the same on a plain SIGTERM. Most deployments will not notice, because Kubernetes SIGKILLs at terminationGracePeriodSeconds. On the fatal-error path there is no such backstop: shutdownForceTime is unset by default (#1072) so forceShutdown$() returns EMPTY, and the Worker waits forever.

Production impact. One Activity doing executeUpdateWithStart stayed outstanding for 137,014,598 ms (38.06 h) with its abort signal already fired. Two Workers processed nothing for ~39 hours and recovered only when an unrelated deploy replaced the pods. The trigger for the underlying OOM was #2227, but the outage length is down to this: the Activity could not be cancelled, so the drain never finished and the process never exited.

Minimal Reproduction

This is a documented-contract violation, verifiable by inspection rather than execution — the three files above are sufficient. To observe it at runtime, in any Activity:

import { getClient, Context } from '@temporalio/activity';

export async function probe(): Promise<void> {
  const client = getClient();                    // NativeConnection-backed
  // A call that will not complete on its own, e.g. awaiting the result of a
  // workflow on a task queue nobody polls:
  const handle = client.workflow.getHandle('some-workflow-nobody-will-run');
  await handle.result();                         // hangs
}

Start this Activity, then shut the Worker down. Context.current().cancellationSignal.aborted becomes true and Context.current().cancelled rejects with CancelledFailure: WORKER_SHUTDOWN, but handle.result() never rejects and the Worker stays in DRAINING. Performing the identical call through a @temporalio/client Connection with an explicit connection.withAbortSignal(sig, …) rejects immediately with gRPC code 1 — the same signal, opposite outcome.

Environment/Versions

• OS and processor: Linux x86_64, containerised
• Temporal Version: Temporal Cloud (exact server build not verifiable from our side); SDK @temporalio/worker 1.21.1 (current latest)
• Are you using Docker or Kubernetes or building Temporal from source? Kubernetes (EKS), official SDK release from npm
• Node 24

Additional context

Suggested fixes, in preference order:

1. Implement the documented behaviour — have NativeConnection.sendRequest observe ctx.abortSignal and cancel the underlying Core request, matching Connection.
2. If Core cannot currently cancel an in-flight request, make the mismatch explicit: correct the JSDoc and consider throwing or warning when withAbortSignal is used on a NativeConnection, so the no-op is not silent.
3. Independently, bound the drain after a fatal error (filed separately as #2266). Even with abort working, a genuinely non-returning Activity still hangs worker.run() forever, which is the state #1539 set out to eliminate ("the Worker's run promise not resolving/failing even though the Worker has reached FAILED state") and which #1536 asked to end by terminating "ASAP". runUntil received a bound for exactly this (promiseCompletionTimeout, defaulting to 0); bare run() did not.

Related work:

• #1739 / PR #2264 — adding ActivityContext.workerShuttingDown. Adjacent and useful, but an opt-in notification; it does not change the getClient() abort, so it would not address this.
• temporalio/sdk-rust#1297 — same user-visible symptom (Worker.run() never returns after SIGTERM) from a different cause (activity poll not resolving, rather than an activity body blocked on an uncancellable call).
• #1072 — established the shutdownGraceTime: 0 / shutdownForceTime: undefined defaults.

Note withDeadline is honoured by NativeConnection (sendRequest reads ctx.deadline), so it is a viable caller-side mitigation and is what we are adopting — but it is not a substitute for cancellation, since a deadline must be guessed up front whereas cancellation is the signal that the work is no longer wanted.

One inaccuracy in the JSDoc while you are in there: NativeConnection.withAbortSignal describes itself as "a convenience wrapper around NativeConnection.withAbortSignal" — a self-reference, presumably copied from Connection.