logpuller: realtime large transactions can OOM before reaching eventstore
@asddongmen is already working on this.
Since Jul 7, 2026.
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
Problem
TiCDC can OOM when it is already running and receives a very large transaction
from the realtime TiKV CDC stream. In one reproduced case, a single 20GiB
transaction caused the CDC process to be killed by cgroup OOM around commit
time. After CDC restarted, the same transaction could be synchronized
successfully.
This issue focuses on the logpuller realtime ingestion path before rows are
persisted into eventstore.
Current logic
In logservice/logpuller/region_event_handler.go, TiKV CDC row events are
handled differently by event type:
Event_COMMITTEDis converted tocommon.RawKVEntryand appended to
span.kvEventsCachedirectly.Event_PREWRITEis stored instate.matcher.putPrewriteRow.Event_COMMITcallsstate.matcher.matchRowto match the cached prewrite
row and fillValue/OldValue.
The matcher in logservice/logpuller/txn_matcher.go keeps unmatched prewrite
rows in unmatchedValue, keyed by (startTs, key).
For a realtime large transaction, CDC can receive many PREWRITE events first
and keep their row values in txnMatcher until the corresponding COMMIT
events arrive. At the same time, memory can also be held by gRPC/protobuf
unmarshal, kvEventsCache, the eventstore write queue, and Pebble batch growth.
For a 20GiB transaction, these holders can overlap and push the process over
the memory limit.
After CDC restarts, the upstream transaction has already committed. The same
history no longer stays in the "realtime PREWRITE waiting for COMMIT" state, so
CDC can continue from committed data without holding the whole transaction in
txnMatcher. This explains why the process can OOM in the realtime path but
successfully finish after restart.
Evidence
Raw pprof profiles are uploaded here:
https://gist.github.com/asddongmen/f9c40dfa1e52166408e3ea84c2144b61
The files are base64 encoded because GitHub gist does not accept binary pprof
files through gh gist create. Restore one with:
base64 -d highfreq-heap-31g-20260703-102906.pb.gz.b64 > highfreq-heap-31g-20260703-102906.pb.gz
go tool pprof -top highfreq-heap-31g-20260703-102906.pb.gz
Key profile observations:
highfreq-heap-31g-20260703-102906.pb.gzgithub.com/pingcap/kvproto/pkg/cdcpb.(*Event_Row).Unmarshal: 8.20GiB flat, 74.18%github.com/pingcap/ticdc/logservice/logpuller.(*regionRequestWorker).receiveAndDispatchChangeEvents: 8.48GiB cumulative, 76.70%github.com/cockroachdb/pebble.(*Batch).grow: 0.56GiB flat, 5.09%
heap-28g-20260703-101449.pb.gzgithub.com/pingcap/kvproto/pkg/cdcpb.(*Event_Row).Unmarshal: 5.93GiB flat, 47.07%github.com/cockroachdb/pebble.(*Batch).grow: 4.25GiB flat, 33.74%github.com/pingcap/ticdc/logservice/logpuller.(*regionRequestWorker).receiveAndDispatchChangeEvents: 6155.65MiB cumulative, 47.72%
Expected behavior
The logpuller realtime path should have bounded memory behavior for very large
transactions. A very large transaction may be slow, but it should not require a
process OOM and restart to finish syncing.
Possible areas to investigate:
- bound or spill
txnMatcherprewrite bytes; - add backpressure based on bytes held by
kvEventsCache/ eventstore write
queue; - avoid building very large Pebble write batches for a single realtime burst.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.