copy: single node performance analysis TODO

We need to develop a deeper understanding of where we are at performance wise before meeting with KV/storage to discuss next steps. This is an issue to track notes, mysteries and TODOs for information gathering to inform those discussions.

There's a bunch of stuff we've tried that hasn't moved the needle:

- Turning off fsync didn't help
- Playing with pebble flush/compaction switches didn't help

Somes notes:

- Disk throughput wise we seem to peak at about 1/10 of max disk throughput
- Performance isn't consistent, as we send batches the initial batches are accepted very quickly and later batches take longer to process, why? Is it just that
- A pure primary index insert w/ no secondary indexes performs much better (duh), but does it come close to saturating disk? The secondary indexes are all non-unique so should be blind puts.
- Why does Mac M1 scale much better with batch size than x86? Is MacOS just lying/cheating? I don't think so...

TODO:

[ ] - Get some up to date CPU profiles. Isolate/compare initial speed and end of insert speed. Also compare x86 vs arm
[ ] - Get memory, locking profiles as well
[ ] - Can we trace what takes critical path goroutines off the CPU somehow? Basically there's the conn_executor goroutine and all the kvsender forks (single node, mulitple ranges involved). I think they mostly just go straight to storage layer and block on read syscalls after first compaction/flush causes the PK index to miss on reads (but why doesn't making memtable bigger help?). And we just insert the PK why is it so fast? Can the go scheduler trace stuff help shine a light here?
[ ] - What does turning off GOGC do?

Jira issue: CRDB-27637

No source files or tests are named; start by collecting the up-to-date CPU, memory, and locking profiles requested in the TODO, comparing initial versus later inserts and x86 versus ARM. Examine the conn_executor, kvsender, storage-layer blocking, and Go scheduler behavior, with documented findings ready for the KV/storage discussion as the definition of done.

Written by the indexing model from the issue text.