least_time: unclamped decay shift misranks long-idle peers (UB shift exponent)

Open
#1,675 1 comment 0 reactions 1 assignee View on GitHub

@sbhowmikf5 is already working on this.

Since Aug 24, 2026.

Assessment

This issue has not been assessed yet.

Description

Bug Overview

ngx_http_upstream_least_time_eta() decays a peer's response time with an unclamped shift (src/http/modules/ngx_http_upstream_least_time_module.c:434):

if (now - peer->checked > peer->fail_timeout) {
    /* once in fail_timeout make response time of a peer 2 times
     * lower to give chances to slow peers */
    rt >>= (now - peer->checked) / (peer->fail_timeout + 1);
}

For a peer with checked == 0 (or one idle long enough), the shift amount can vastly exceed the width of rt. Shifting by >= the type width is undefined behaviour — and beyond the formal UB, x86 masks the shift count mod 64, so instead of the comment's intended decay-toward-zero, rt becomes an arbitrary intermediate value and the balancer misranks the peer. This affects non-sanitised builds: a long-idle or never-checked peer gets a wrong ETA whenever the shift amount reaches 64 (e.g. any idle period past 64 × (fail_timeout + 1) seconds — or immediately when checked is 0 and the division leaves epoch-scale seconds).

Under UBSan (-fno-sanitize-recover=undefined) it is also a fatal trap; observed exponent 1787288950 (epoch seconds with fail_timeout=0).

Expected Behavior

The decay saturates: once the idle period implies a shift of the type width or more, rt becomes 0 ("give this peer a chance"), matching the comment's intent — with no UB on the way there.

Steps to Reproduce the Bug

Using the sibling test suite, with an UBSan build (same configure as #1671):

./configure --with-debug \
  --with-cc-opt="-fsanitize=undefined -fno-sanitize-recover=undefined -O1 -g" \
  --with-ld-opt="-fsanitize=undefined" && make -j"$(nproc)"
git clone --depth 1 https://github.com/nginx/nginx-tests.git
cd nginx-tests
TEST_NGINX_BINARY=/path/to/objs/nginx prove upstream_least_time.t

Fails 5/14 with workers exiting per request (requires the #1672 fix first, or the harness's debug logging kills the worker before this path is reached).

NGINX Configuration

Driven by nginx-tests/upstream_least_time.t (an upstream with least_time and fail_timeout=0 peers). Any least_time upstream whose peers can sit idle reproduces the misranking arithmetic on regular builds.

NGINX version and build configuration options

nginx version: nginx/1.31.4
built by gcc 15.3.1 20260722 (Red Hat 15.3.1-1) (GCC)
configure arguments: --with-debug --with-cc-opt='-fsanitize=undefined -fno-sanitize-recover=undefined -O1 -g' --with-ld-opt=-fsanitize=undefined

Also present in current master (the function is unchanged).

Environment where NGINX is being built and/or deployed

Fedora 43 (WSL2), gcc 15.3.1, glibc 2.42. The UB/misranking is build-independent; the sanitizer trap follows the flags.

Architecture

Linux 6.6.87.2-microsoft-standard-WSL2 x86_64 GNU/Linux

NGINX Debug Log

src/http/modules/ngx_http_upstream_least_time_module.c:434:12: runtime error: shift exponent 1787288950 is too large for 64-bit type 'long unsigned int'

(worker exits with code 1 and is respawned per request under -fno-sanitize-recover)

Additional Context

One of four instances found by running nginx-tests under UBSan once #1672 made that possible (see #1671's "Additional Context" for the inventory); filed first of the four because the misranking is observable without sanitizers. Suggested shape, saturating per the comment's intent:

if (now - peer->checked > peer->fail_timeout) {
    shift = (now - peer->checked) / (peer->fail_timeout + 1);
    rt = (shift < 8 * sizeof(ngx_msec_t)) ? rt >> shift : 0;
}

Happy to submit a PR.

Dominant language
C
Stars
31.7k
Forks
8.3k
Avg merge
1d 12h
Merged PRs (30d)
13

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nginx/nginx

All issues in nginx/nginx

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.