temporalio / temporalio/temporal
Replication tasks not cleaned up on source when source/target shard counts differ (pull replication)
Open
@yux0 is already working on this.
Since Jun 2, 2026.
potential-bug
teams/cgs
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
Expected behavior
In pull-based replication (history.enableReplicationStream = false), after the standby acks task ID X, the source's per-shard cleanup loop should delete acked replication tasks (type = 4 rows in executions), in case the standby cluster has twice the number of shards as the source cluster
Actual behavior
When the target cluster has twice the number of shards as the source, replication tasks on the source cluster are never deleted, even after:
- The standby has fully applied all generated replication tasks.
- A namespace handover successfully promotes the standby to active.
All replication tasks in source cluster accumulate indefinitely. No error is logged.
Steps to reproduce
- Set up two clusters with global replication:
- Cluster A (source):
history.numberOfShards = 8192. - Cluster B (standby):
history.numberOfShards = 16384(double A's numberOfShards) history.enableReplicationStream = false(default).- Both clusters are
Enabled = truein the other's cluster metadata.
- Cluster A (source):
- Register a global namespace replicated from A to B.
- Run workflows on A; let B apply the tasks.
- Wait several
history.ReplicationTaskProcessorCleanupIntervalcycles (default 1 minute). - Query Cassandra:
SELECT count(*) FROM executions WHERE shard_id = <s> AND type = 4 .... Count grows monotonically, never shrinks.
Specifications
- Version: Temporal Server 1.22.0
- Platform: Kubernetes
- Persistence store: ScyllaDB 5.2 and ScyllaDB 2024.1
- Source code where reader states are written in source cluster's shards: https://github.com/temporalio/temporal/blob/main/service/history/api/replication/get_tasks.go#L45
- Public document explicitly confirms that replication should work when History shard counts must be a multiple of two: https://docs.temporal.io/cloud/migrate/automated#:~:text=History%20shard%20counts%20must%20be%20a%20multiple%20of%20two.
- Shard info of a shard that has replication tasks not yet cleaned in source cluster:
$ tdbg shard describe --shard-id 19
{
"shardId": 19,
"rangeId": "187",
"owner": "10.0.73.95:7234-108-15476a71-98f8-443f-998f-c530e0db169c",
"updateTime": "2026-05-07T06:24:20.449962317Z",
"queueStates": {
"1": {
"exclusiveReaderHighWatermark": {
"fireTime": "1970-01-01T00:00:00Z",
"taskId": "196681394"
}
},
"2": {
"exclusiveReaderHighWatermark": {
"fireTime": "2026-05-07T06:23:39.096Z"
}
},
"3": {
"readerStates": {
"12884901907": {
"scopes": [
{
"range": {
"inclusiveMin": {
"fireTime": "1970-01-01T00:00:00Z",
"taskId": "196681394"
},
"exclusiveMax": {
"fireTime": "1970-01-01T00:00:00Z",
"taskId": "9223372036854775807"
}
},
"predicate": {
"predicateType": "Universal"
}
}
]
}
}
},
"4": {
"exclusiveReaderHighWatermark": {
"fireTime": "1970-01-01T00:00:00Z",
"taskId": "196681394"
}
}
}
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.