firecracker-microvm / firecracker-microvm/firecracker

[Bug] Rate limiter can permanently stall on sub-millisecond refill

Open
#6,212 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
36.7k
Forks
2.6k
Avg merge
3d 3h
Merged PRs (30d)
58

Description

# Describe the bug

The rate limiter can permanently stall when a request exceeds the currently available tokens and the calculated refill duration is less than 1 millisecond.

In this case, the calculated duration is truncated to zero when converted to milliseconds. The timerfd is then disarmed, while the rate limiter still considers its timer active. As a result, no further refill notification is generated and the rate limiter remains blocked indefinitely.

## To Reproduce

The issue can be reproduced deterministically with a bandwidth rate limiter configured with:

* `size = 4095`
* `refill_time = 1`
* `one_time_burst = 0`

and a request consuming 4096 bytes.

For example, the relevant configuration is conceptually:

```json
{
"bandwidth": {
"size": 4095,
"one_time_burst": 0,
"refill_time": 1
}
}
```

When a 4096-byte request is processed, the request consumes more tokens than are currently available.
The rate limiter calculates a refill duration for the outstanding token debt. The resulting duration is below 1 millisecond and is truncated to `0ms`.
This causes the timerfd to be disarmed, while the rate limiter's timer state remains active.
After this point, the rate limiter does not receive the expected timer event and remains blocked indefinitely.
A regression test and proposed fix are available in #6211.

## Expected behaviour
The rate limiter should eventually refill the required tokens and allow the blocked request to proceed.
A sub-millisecond refill interval should not result in the timer being permanently disarmed.
## Environment

* Firecracker version: main (1.18.0-dev)
* Host and guest kernel versions:
- Host: 5.15.0-176-generic (Ubuntu 22.04.3 LTS, x86_64)
- Guest: 5.10.225 (Firecracker CI guest kernel, x86_64)
* Rootfs used: minimal ext4 rootfs (rootfs_shell.ext4) with a static C
/init that execs busybox sh, built via mkfs.ext4 -d
* Architecture: x86_64
* Any other relevant software versions: N/A

## Additional context
This was found while testing the Firecracker rate limiter with a small bucket size and refill interval.
The issue is deterministic and can cause a rate-limited device request to remain blocked indefinitely.
The suspected root cause is the conversion of the calculated refill duration to milliseconds, which truncates sub-millisecond durations to zero before arming the timerfd.
A proposed fix is available in #6211, including regression coverage.

## Checks

* [x] Have you searched the Firecracker Issues database for similar problems?
* [x] Have you read the existing relevant Firecracker documentation?
* [x] Are you certain the bug being reported is a Firecracker issue?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing issue #6211, which contains the proposed fix and regression coverage, then reproduce the deterministic configuration with size 4095, refill_time 1, and one_time_burst 0. Done means the 4096-byte request eventually proceeds and the regression test covers sub-millisecond refill handling without permanently disarming the timerfd.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.