temporalio / temporalio/temporal

Nexus: server may send malformed `request-timeout` header (negative values and units outside the Nexus grammar)

Open
#11,569 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Expected Behavior

The request-timeout header on a dispatched Nexus task should always match the grammar the server itself parses these durations with — no sign, and only the ms, s, and m units:

https://github.com/temporalio/temporal/blob/7b19e18a7d1e33254a97f226686654fc6f67ebf5/common/nexus/nexusrpc/api.go#L254

var durationRegexp = regexp.MustCompile(`^(\d+(?:\.\d+)?)(ms|s|m)$`)

Actual Behavior

Matching formats the header with Go's time.Duration.String(), unclamped:

https://github.com/temporalio/temporal/blob/7b19e18a7d1e33254a97f226686654fc6f67ebf5/service/matching/matching_engine.go#L2785-L2789

nexusReq.Header[nexus.HeaderRequestTimeout] = time.Until(task.nexus.deadline).String()
// Java SDK currently expects the header in this form. We should be able to remove this duplication sometime mid 2025.
nexusReq.Header["Request-Timeout"] = time.Until(task.nexus.deadline).String()
if !task.nexus.operationDeadline.IsZero() {
    nexusReq.Header[nexus.HeaderOperationTimeout] = commonnexus.FormatDuration(time.Until(task.nexus.operationDeadline))
}

That can produce values the grammar does not allow:

  • units below the millisecond (88.458µs), above the minute (2h), and multi-unit values (1m30s);
  • negative values, when the task's deadline has already elapsed by the time it is matched to a poller (-88.458µs).
  • the value 0.

Note that the operation-timeout header on the very next line already goes through commonnexus.FormatDuration, which is grammar-compliant. The request-timeout header never got the same treatment, and the line is unchanged since #5844.

Impact

An SDK that implements the documented grammar rejects the value and silently loses the request timeout for that task, so the handler is never told the request has been abandoned.

The negative case was observed with the Rust SDK against a custom dev server (CLI v1.7.4-standalone-nexus-operations, which contains server v1.32.0-158.0), which dispatched tasks a few microseconds past their deadline: -88.458µs, -70.416µs, -158.375µs. Ten local runs against CLI 1.8.x (server v1.31.2) did not reproduce the negative timeouts issue (the formatting issue is still present), but I strongly believe that's due to other factors that influence performance of the server (so we never hit the race condition that is required to trigger this negative timeout issue in my test setup) rather than an actual fix of the bug.

Suggested Fix

  • Clamp the remaining time at zero and format it with commonnexus.FormatDuration for both the request-timeout and legacy Request-Timeout headers. Clamping is needed in addition to the formatting change, since Duration.Milliseconds() truncates toward zero and would still yield e.g. -1ms for a 1.5ms overshoot.

  • Matching should probably never dispatch a task whose deadline has already elapsed at all (i.e. where value would be less than 1ms).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in service/matching/matching_engine.go at the request-timeout and legacy Request-Timeout header assignments, then read commonnexus.FormatDuration and the duration grammar in common/nexus/nexusrpc/api.go. Done means both headers use grammar-compliant, non-negative values even when the task deadline has elapsed, with dispatch behavior for already-expired tasks considered separately.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.