tikv / tikv/pd

Bound request-driven memory growth and GC amplification

Open
#11,164 0 comments 0 reactions 0 assignees View on GitHub
type/development
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Development Task

### Background

On large clusters, several PD request paths can allocate or retain memory in proportion to request or heartbeat backlog. Once the heap and pointer-rich live objects grow, GC mark/assist consumes CPU, request processing slows, queues grow further, and a positive feedback loop can form:

`processing slows -> backlog grows -> heap/object count grows -> GC CPU grows -> processing slows further`

Audit baseline: `upstream/master` at `56d3fe03a42fa2e70836480c97a1c8013356063c` (2026-08-18).

This is an umbrella issue. Some items are confirmed incidents; others are code-audit risks that need load-test/profile validation. Not all are leaks: several are deliberately unbounded or very-high-bounded queues, full-result materialization, or large live pointer graphs.

### High-risk paths

1. **Classic PD streaming gRPC can buffer an unbounded request body**
- Classic PD multiplexes HTTP and gRPC through embedded etcd and `grpc.Server.ServeHTTP` (`server/server.go`).
- grpc-go's handler transport reads the HTTP request body into an unbounded receive buffer without application `Recv()` backpressure.
- Streaming RPCs such as `RegionHeartbeat` can therefore retain memory proportional to a slow consumer's backlog. Standalone microservices using native `grpc.Server.Serve` do not use this handler transport.
- Related: #9649 and https://github.com/grpc/grpc-go/issues/7261.

2. **HotCache heartbeat work queues have multi-million-item ceilings**
- `pkg/statistics/hot_cache.go` creates two queues with `chanMaxLength = 6,000,000` and one consumer per queue.
- A region heartbeat can enqueue multiple closures from `pkg/cluster/cluster.go`; queued closures retain `RegionInfo` and related objects.
- There is no latest-per-region coalescing, so obsolete work can remain live during sustained backlog.

3. **ConcurrentRunner can retain up to 20 million tasks per runner**
- `pkg/ratelimit/runner.go` uses `maxPendingTaskNum = 20,000,000`; classic PD creates four runners.
- Tasks are allocated before deduplication, retained tasks bypass the normal age-based drop, and peak map/slice capacity remains after a burst.

4. **Heavy APIs materialize request-sized results without safe default bounds**
- `ScanRegions` accepts an unlimited result when `limit <= 0`; the deprecated response also builds parallel region/leader slices.
- `/regions` obtains all region pointers and serializes one complete JSON response.
- `/hotspot/regions/history` has no pagination/limit and materializes the complete result.
- Region full sync calls `GetRegions()`, which alone allocates an N-pointer slice (about 800 MiB for 100 million regions on 64-bit systems).
- The default gRPC rate-limit configuration is empty, so these expensive requests have no safe concurrency ceiling. Related: #8882.

5. **HTTP middleware amplifies large request bodies**
- Audit/rate-limit middleware reads the complete body with `io.ReadAll`, converts it to `string`, restores it, and handlers commonly read it again.
- Production request boundaries do not consistently use `http.MaxBytesReader`.
- Resource-manager config handling recently removed its 1 MiB body limit in #11145, then parses and rebuilds JSON.

6. **Recurring full snapshots create avoidable allocation churn**
- Keyspace-group state is fully reloaded every second, causing etcd reads, JSON allocations, and GC pressure. The event-driven fix is tracked by #11026 / #11027.

7. **Large baseline caches magnify GC cost**
- `RegionsInfo` maintains several B-tree/map indexes over region pointers. Most indexes share objects, but the aggregate live pointer graph is large and expensive for GC to scan.
- At 100 million regions, profiling in #10773 showed CPU dominated by GC, with region metadata/protobuf/B-tree objects among the largest heap users.
- Optional per-region placement-rule and hot-history caches add further retention. Related: #5864 and #5713.

8. **Per-store metric state can outlive removed stores**
- Stale per-store metric vectors/maps are tracked by #11126 / #11127.

`server-memory-limit` defaults to disabled, and changing GOGC cannot reclaim reachable queued/cached objects, so GC tuning is not a root fix for these paths.

Contributor guide

Open the contributing guide

Research direction

Start by splitting the umbrella into one path at a time, beginning with server/server.go and the queues in pkg/statistics/hot_cache.go and pkg/ratelimit/runner.go. Review the related issues and validate each suspected risk with load tests or memory/CPU profiles. Done means each selected path has an agreed bound or lifecycle fix and evidence that backlog and GC amplification are controlled.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
backend, distributed-systems, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.