pingcap / pingcap/ticdc

logpuller: realtime large transactions can OOM before reaching eventstore

Open
#5,561 0 comments 0 reactions 1 assignee View on GitHub

@asddongmen is already working on this.

Since Jul 7, 2026.

component/puller type/enhancement
Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

Problem

Image

TiCDC can OOM when it is already running and receives a very large transaction
from the realtime TiKV CDC stream. In one reproduced case, a single 20GiB
transaction caused the CDC process to be killed by cgroup OOM around commit
time. After CDC restarted, the same transaction could be synchronized
successfully.

This issue focuses on the logpuller realtime ingestion path before rows are
persisted into eventstore.

Current logic

In logservice/logpuller/region_event_handler.go, TiKV CDC row events are
handled differently by event type:

  • Event_COMMITTED is converted to common.RawKVEntry and appended to
    span.kvEventsCache directly.
  • Event_PREWRITE is stored in state.matcher.putPrewriteRow.
  • Event_COMMIT calls state.matcher.matchRow to match the cached prewrite
    row and fill Value / OldValue.

The matcher in logservice/logpuller/txn_matcher.go keeps unmatched prewrite
rows in unmatchedValue, keyed by (startTs, key).

For a realtime large transaction, CDC can receive many PREWRITE events first
and keep their row values in txnMatcher until the corresponding COMMIT
events arrive. At the same time, memory can also be held by gRPC/protobuf
unmarshal, kvEventsCache, the eventstore write queue, and Pebble batch growth.
For a 20GiB transaction, these holders can overlap and push the process over
the memory limit.

After CDC restarts, the upstream transaction has already committed. The same
history no longer stays in the "realtime PREWRITE waiting for COMMIT" state, so
CDC can continue from committed data without holding the whole transaction in
txnMatcher. This explains why the process can OOM in the realtime path but
successfully finish after restart.

Evidence

Raw pprof profiles are uploaded here:

https://gist.github.com/asddongmen/f9c40dfa1e52166408e3ea84c2144b61

The files are base64 encoded because GitHub gist does not accept binary pprof
files through gh gist create. Restore one with:

base64 -d highfreq-heap-31g-20260703-102906.pb.gz.b64 > highfreq-heap-31g-20260703-102906.pb.gz
go tool pprof -top highfreq-heap-31g-20260703-102906.pb.gz

Key profile observations:

  • highfreq-heap-31g-20260703-102906.pb.gz
    • github.com/pingcap/kvproto/pkg/cdcpb.(*Event_Row).Unmarshal: 8.20GiB flat, 74.18%
    • github.com/pingcap/ticdc/logservice/logpuller.(*regionRequestWorker).receiveAndDispatchChangeEvents: 8.48GiB cumulative, 76.70%
    • github.com/cockroachdb/pebble.(*Batch).grow: 0.56GiB flat, 5.09%
  • heap-28g-20260703-101449.pb.gz
    • github.com/pingcap/kvproto/pkg/cdcpb.(*Event_Row).Unmarshal: 5.93GiB flat, 47.07%
    • github.com/cockroachdb/pebble.(*Batch).grow: 4.25GiB flat, 33.74%
    • github.com/pingcap/ticdc/logservice/logpuller.(*regionRequestWorker).receiveAndDispatchChangeEvents: 6155.65MiB cumulative, 47.72%

Expected behavior

The logpuller realtime path should have bounded memory behavior for very large
transactions. A very large transaction may be slow, but it should not require a
process OOM and restart to finish syncing.

Possible areas to investigate:

  • bound or spill txnMatcher prewrite bytes;
  • add backpressure based on bytes held by kvEventsCache / eventstore write
    queue;
  • avoid building very large Pebble write batches for a single realtime burst.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.