In-Memory Relaylog

This proposal is authored by the AWS RDS Aurora MySQL team. We were encouraged by the Change Stream Applier (CSA) work Oracle presented on April 29, and in this discussion we propose optimizations to further improve CSA's replication performance. We are looking to collaborate with the community and implement these improvements in upstream MySQL. The tech contact is Xueting Wu (@ams-binlog-replication).

Introduction

Oracle recently introduced a MySQL Labs Release featuring the Change Stream Applier (CSA), a new execution model for the MySQL logical replication applier that significantly improves SQL thread parallelism and throughput. At AWS Aurora MySQL, we have been independently implementing equivalent optimizations in Binlog, including parallelizing the deserialization of binlog events, improving scheduling by enabling worker threads to make partial progress during transaction execution, and parallelizing the application of large transactions, work that overlaps considerably with CSA's current and future scope.

We have conducted experiments with CSA and observed throughput improvements in the SQL subsystem (coordinator + worker threads) of up to 2×. We believe additional optimizations can push IO thread performance to 2x, and SQL thread by 1.5×. This document presents those ideas for your consideration.

CSA Performance and Bottlenecks

We benchmarked CSA against the baseline MySQL 9.7's multi-threaded applier replication under identical workload conditions. All configurations used 64 worker threads, GTID enabled, binary logging disabled on the replica, and flush-at-commit (flush_trx = 1). Commit order and performance schema were disabled on the replica to reduce overhead.

Throughput metric: We generate a sustained Sysbench workload on the source and measure the rate of binlog bytes processed per second (MB/s) at each stage of the replication pipeline. To measure IO throughput, we pause the SQL thread and let the IO thread fill the relay log unimpeded. To measure SQL throughput, we pause the IO thread after the relay log has been populated and measure how long the SQL subsystem takes to process the buffered events. E2E throughput is the effective replication rate from source to committed replica state with both threads running concurrently.

Configuration	IO Throughput	SQL Throughput	E2E Throughput
MySQL 9.7 Source	N/A	N/A	56 MB/s
MySQL 9.7 Replica – no CSA	40 MB/s	23 MB/s - 24.4 MB/s	22.2 MB/s
MySQL 9.7 Replica - CSA	40 MB/s	40 – 47.3 MB/s	36.6 - 39.7 MB/s

We draw the following conclusions from the above benchmarks:

• CSA improves SQL subsystem performance by nearly 2×, raising SQL throughput from 23–24.4 MB/s to 40–47.3 MB/s.
• The replica still cannot keep pace with the source. The source generates binlog events at 56 MB/s, but the best E2E replica throughput with CSA is 39.7 MB/s—a 29% gap that leads to growing replication lag under sustained load.
• The IO thread is now the bottleneck. At 40 MB/s, the IO thread caps E2E throughput regardless of SQL-side improvements. To fully leverage CSA's gains, the IO thread must exceed 40 MB/s.

Proposals

We have analyzed CSA's design in conjunction with what we were planning to build at Aurora MySQL and have identified two ideas that can significantly improve CSA's performance.

Design Proposal 1: In-Memory Relay Log

In the current architecture, the IO thread writes incoming binlog events to a relay log on persistent storage, and the SQL threads read them back from disk. This write-and-read round trip introduces latency that varies with the underlying storage type and becomes the dominant cost on slower volumes.

We propose replacing the on-disk relay log with a pre-allocated in-memory structure. The IO thread writes events directly into memory, and the SQL threads read from the same structure—eliminating the storage round trip entirely.

Aurora MySQL has already implemented this optimization: In-Memory Relay Log. In our experience, the in-memory relay log improves IO thread performance by 2×, pushing throughput beyond 60 MB/s. Additionally, it also marginally improves SQL thread performance, by eliminating relay log writes to storage, overall I/O pressure on the system is reduced, making downstream commits at the worker threads faster.

Bypassing the on-disk relay log is safe when using GTID with auto-positioning, which CSA mandates. The engine does not read the relay log during recovery, it simply requests missing transactions from the source based on the GTID gap. This works regardless of whether replica_preserve_commit_order is ON or OFF. Our Aurora MySQL customers have been using this feature reliably for over a year.

Design Proposal 2: Group Boundary Queue

In the current CSA architecture, the SQL coordinator thread reads events from the relay log to identify transaction boundaries, determine group membership, and calculate dependency information before dispatching work to applier threads.

This means the coordinator must sequentially scan and partially deserialize every event—work that is redundant when the IO thread has already parsed these same events during reception. We propose that the IO thread, while writing events to the in-memory relay log, additionally collects transaction metadata and pushes it into a lightweight in-memory FIFO structure we call the Group Boundary Queue (GBQ). Metadata is per group (transaction) includes transaction boundaries (begin/end positions), transaction size, and dependency information already available in the binlog (e.g., last_committed, sequence_number).

The SQL coordinator then pops entries from the GBQ instead of scanning the relay log. With boundaries and dependency information readily available, the coordinator can immediately perform scheduling decisions and dispatch work to applier threads, which read event data from the in-memory relay log using the positions provided by the GBQ entry.

This has multiple benefits: the coordinator thread now reads groups instead of individual events, it no longer needs to deserialize events, and it reduces I/O contention by avoiding a single I/O bottleneck where both the coordinator and worker threads compete to read from the same relay log.

Conclusion

We believe CSA is a major step in the right direction, and that there is still room for further improvement. Together, the two proposals advanced by AWS can improve IO thread performance by 2× and SQL thread performance by 30% over CSA.

We are looking to define a collaborative path forward on these projects. We can port the in-memory relay log code for CSA and contribute it to Oracle, and we are equally open to Oracle implementing it independently. Regardless of approach, there is a clear opportunity to improve replication performance so that the replica can keep pace with the binlog source.

Start with the issue's CSA performance discussion and the two proposals: In-Memory Relay Log and Group Boundary Queue. Review the linked Aurora MySQL documentation for the existing in-memory approach, then establish an agreed upstream design and implementation scope; done would require a collaborative path forward rather than a single defined edit or test.