cockroachdb / cockroachdb/cockroach

crosscluster: progress fan-in erases panics and tracing-aggregator update is not deferred

Open
#174,954 2 comments 0 reactions 0 assignees View on GitHub
A-disaster-recovery C-bug O-agent T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Summary:**

The same fan-in and cleanup shapes found in backup/restore are present in the cross-cluster replication paths. Part of #174944.

**Findings:**

- `physical.startDistIngestion` progress fan-in: `tracingAggCh` — a context-guarded consumer loop paired with an unguarded producer send, so a consumer panic is swallowed by `ctxgroup.GoCtx` while the producer parks on a bare channel send forever. The panic is erased and the PCR job hangs with no log line. Live today.
- `physical.streamIngestionResumer` tracing-aggregator update: the `mu.Lock(); map[k] = *ptr; mu.Unlock()` sequence is not deferred, while `logical.logicalReplicationResumer` defers the identical logic.
- `ctxgroup.Group.Go` used instead of `GoCtx` in `replicationutils` — `Go` is a bare pass-through to `errgroup` with no recover at all, so a panic in one of those goroutines is invisible to any recovery boundary and kills the node regardless of how one is designed.

**Next Steps:**

- [ ] Make the `tracingAggCh` producer send select on `ctx.Done()`
- [ ] Defer the tracing-aggregator update to match `logicalReplicationResumer`
- [ ] Convert the `replicationutils` `.Go` calls to `.GoCtx`, or document why not

Epic: none

Jira issue: CRDB-68125

Contributor guide

Open the contributing guide

Research direction

Start with physical.startDistIngestion and physical.streamIngestionResumer, then compare their fan-in and tracing-aggregator handling with logical.logicalReplicationResumer. Review the replicationutils ctxgroup.Go calls and the GoCtx behavior. Done means the three listed cleanup and panic-handling concerns are addressed or documented, with tests covering the affected paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.