crosscluster/physical: investigate bulk ops bottlenecks

Currently, the replication lag can spike to O(2*bulk job runtime) after a backfill or import. We should address this. Here's our current understanding of the problem:
- the source side bulk operations issue a ton of manual split and scatter operations to the key space the bulk operation is about to ingest into. While, this trick significantly increases Import or backfill performance by preventing the allocator from issuing its own splits and rebalance operations on non-empty ingesting data, the destination side of PCR is completely unaware of these splits-- we only replicate key data, not range placement data. So, destination side PCR sees a ton of ingesting data into what was previously empty key space, causing the dest cluster allocator to issue these expensive snapshots, slowing down ingestion.
- Before implementing better replanning and fine grain checkpointing during catchup scans, PCR replication would essentially grind to a halt, because PCR didn't know how to replan and redistribute work during one of these ingestion events. In other words, before those changes, n source side nodes would work on a backfill and 1 dest side node would replicate them. Today, PCR will replan and redistribute work on the destination side if it detect uneven work distribution. With these changes, we've observed replication lag max out around 2x the runtime of the source side bulk operation.

To further reduce lag, we should:
- explore how replicating manual splits from source to dest could speed this up. PCR actually has this ability (code [ref](https://github.com/cockroachdb/cockroach/pull/123225)) but it is currently turned off because our initial benchmarks did not show much promise. We did not understand why yet.
- understand PCR perf on single node clusters. When PCR streams from a single node source to a single node dest, PCR can keep up! But only sometimes. I'd like to understand why single node PCR is so much better at replicating bulk ingests, and further, under what conditions does it fall behind.
- understand the perf impact of reenabling async flushing on range splits
- if all else fails, explore computing replication lag on source side online indexes only

If we cannot reduce lag, in the current PCR design, we could consider tracking source side online indexes and waiting to flip the on switch on the bulk ingested keyspace until PCR catches up. The idea is technically possible for import and restore, but I'd need to double check on index backfills. Before the backfilled index becomes available, I believe we dual write the foreground traffic to both the old and new index, so this blocking strategy would lead to more dual writes, but I guess that is fine?

That being said, this strategy runs against two of PCR's design principles:
- We've designed PCR such that the source app virtual cluster is completely unaware that it is being replicated. This strategy would essentially double the runtime of backfills/imports/restores on the source cluster, and would require a new app to system tenant RPC along the lines of "hey, am i being replicated?"
- PCR runs on system tenants and replicates opaque bytes of app tenants. In other words, PCR is unaware of the source cluster's descriptor state. To implement this idea, the source side system tenant would need to peer into app tenant descriptor state to build the set of spans that are "online". We generally don't like it when system tenants peer into app tenant key space.

If I had a month or 2 to work on this problem, I would first investigate our current bottlenecks as I outlined above before turning to your strategy.

Jira issue: CRDB-51576

Epic CRDB-37540