apache / apache/brpc

[Feature Request] Run-to-Completion (RTC) Worker Loop Support

Open
#3,212 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.6k
Forks
4.1k
Avg merge
2d 12h
Merged PRs (30d)
69

Description

# [Feature Request] Run-to-Completion (RTC) Worker Loop + User Hooks + Optional io_uring Backend

## Background

We run brpc in a **per-thread-per-core** deployment where worker threads are CPU-pinned and also interact with a high-performance I/O stack (SPDK or io_uring). In this model, there are scheduling “gaps” inside the worker loop (e.g., after running a batch of runnable bthreads and before the worker blocks in `epoll_wait`). Those gaps are ideal to execute small, non-blocking work **on the same worker thread** to preserve locality and avoid cross-thread handoff.

Concrete examples of work we want to run in these gaps:
- Harvest/poll **SPDK** completions (poll-mode style)
- Drain **io_uring** CQEs and advance state machines
- Convert completions into runnable units and **push them back to the worker’s local queue**

Today, integrating these cleanly requires extra polling threads, periodic timers, or cross-thread scheduling, which increases tail latency and adds contention.

## Problem

1. **No first-class “run user logic on the current worker thread” hook points**
- Users cannot reliably execute a lightweight function at well-defined points in the worker loop (before/after waiting, between batches, etc.).

2. **No supported way to reschedule derived work back into the *local* run queue**
- For completion-driven workloads, the critical requirement is: harvest completions on the worker thread and immediately enqueue follow-up work to the **same worker’s local queue**. Global queues/cross-thread wakeups increase contention and latency.

3. **epoll-centric waiting is not ideal for io_uring/SPDK-centric stacks**
- For io_uring, batching submit/completion is the natural model and can reduce syscalls.
- For SPDK, polling is the natural model and requires a bounded busy loop or hybrid strategy.

## Requirements

### R1. Run-to-Completion (RTC) style worker execution
Provide an opt-in worker model that repeatedly:
- runs runnable bthreads until a budget is reached,
- runs registered “maintenance” hooks (budgeted),
- waits for events (blocking/polling/hybrid),
- repeats.

### R2. epoll polling / hybrid mode
Support configurable event-wait behavior for the epoll backend:
- **blocking** (default): `epoll_wait(timeout > 0)`
- **polling**: `epoll_wait(timeout = 0)` in a bounded busy loop
- **hybrid**: poll for `N` iterations or `T` microseconds, then fall back to blocking

### R3. User hook registration on worker threads
Provide an API to register callbacks that execute **on the current brpc worker thread** at defined hook points. Hooks must be:
- non-blocking (no sleeping, no long syscalls)
- budgeted (time/ops limited)
- safe to run frequently

### R4. Ability to enqueue work back to the worker’s *local* queue
Hooks must be able to take harvested completions and schedule follow-up work back to the worker’s local run queue (not global), e.g.:
- enqueue an existing bthread/task handle locally
- or submit a closure that is executed by the same worker

### R5. Optional io_uring backend replacing epoll
Support selecting an **io_uring** event backend as an alternative to epoll (opt-in), to:
- reduce syscalls via batching
- align with per-thread-per-core (one ring per worker)
- enable future enhancements (multishot, buffer rings) without forcing epoll

Contributor guide

Open the contributing guide

Research direction

Start by tracing the worker loop, its epoll wait path, and the local run queue. The design must cover opt-in RTC execution, budgeted worker hooks, blocking/polling/hybrid waits, local rescheduling, and an optional io_uring backend; done means these requirements are supported without forcing existing users off epoll.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, backend-api-design, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.