Batched INSERTS with poor locality have poor performance due to waiting on disk reads.

This is the tracking issue to address the issues uncovered by https://github.com/dolthub/dolt/issues/6536

Steps to reproduce:

Run each of the two attached scripts in an empty Dolt repo. You can verify that they have the same result by running them in two different branches and diffing the branches. Yet, the "sorted" script runs approximately 10 times faster than the "unsorted" script.

In MySQL, these two scripts are both as fast as the "sorted" script is in Dolt.

**What's happening:**

The Dolt engine computes a new root hash after every statement that modifies the database. This requires flushing the write session and rechunking the prolly tree for each index. Each time we do this, we invalidate one or more chunks and create new ones. The newly created chunks are stored alongside the old ones in an in-memory table. As a result, the in-memory table grows faster than the actual data being inserted into the database.

Eventually the memory table exceeds its max capacity and all of its chunks get written to disk, and we get a new, empty memory table. Whenever the engine needs one of those chunks (either to check for duplicate keys, or to rechunk it), we load it back from disk into memory.

If the script being executed has good locality, the impact of this is minimal: once a chunk is written to disk, the odds are very low that we will need to read it again.

But if the script being executed has poor locality, then the performance cost can be severe: most chunks will typically end up needing to be read back in from disk.

Because of how many "junk chunks" that this process generates (chunks that are not in the final tree and never need to be read again after being written to disk), this problem occurs even when the memory table is larger than the final size of the database.

The end result: when populating an initially empty database, we spend the majority of the runtime reading from disk. This feels silly.

Our best practices page outlines steps that users can take to mitigate this behavior. From https://docs.dolthub.com/guides/import#import-best-practices:

1. Import one table at a time.
2. Prefer fewer INSERT statements that each insert multiple values over more, shorter statements that only insert a single value each.
3. Sort your inserts by primary key. If a table doesn't have a primary key, consider adding one.

But ideally customers shouldn't need to worry about this. A standard practice query that runs quickly in MySQL should be reasonably fast in Dolt.

**What we can do about it**

We could reduce the amount of work that needs to be done by automatically batching inserts (eliminating the need for suggestion 2 above). But this is just compensating for the fact that rechunking the prolly tree produces so many junk chunks that clutter up the memory table. We want to be able to remove these chunks from the memory table once they're no longer needed, without flushing the table entirely. Keep the chunks in memory that we'll need again, and drop the ones we don't. In the best case we "in-flight garbage collect" these chunks and never write them out to disk. But if this proves to be error-prone, it's fine for us to write those chunks to disk first (and then manually garbage collect them later.) The important part is that we remove them from the memory table.

The biggest obstacle is that the chunk store doesn't know anything about prolly trees or root hashes. On the server, it's also a shared global resource that may be used concurrently.

**Suggestion 1: Reference Counting**

We could have an interface where the application layer can tell that chunk store that it no longer needs a certain chunk. This could be a simple "DropChunk" function that marks chunks as low-priority, but it could also assign a reference count to each chunk: adding a chunk to the store, or requesting a chunk from the store increases its reference count, and dropping the chunk decreases its reference count.

Then, when rechunking indexes, we tell the chunk store to drop the invalidated chunks. When the mem table reaches max capacity, chunks that are marked for dropping are flushed first. Then, if the mem table is still above a certain capacity, the remaining chunks are also flushed.

**Suggestion 2: On-the-fly garbage collction**

This is based on the principle that if a chunk is in the memory table, its parent chunks are almost certainly also in the table (at least, I can't think of any way for this to be invalidated.)

When we need to flush the memory table, we first walk a forest, marking root chunks that we care about and their children in the memory table. We don't load any chunks from disk during this. Then we keep the marked chunks and flush the rest. Then, if the mem table is still above a certain capacity, the remaining chunks are also flushed.

We would need to identify places where this is safe to do, when there's no concurrency happening.

Start by reproducing the difference between the attached sorted and unsorted INSERT scripts in an empty Dolt repo, then read the chunk store, memory table, and prolly-tree rechunking paths described here. Done means reducing unnecessary disk reads and junk chunks for poor-locality inserts while preserving correctness and safe concurrent use.

Written by the indexing model from the issue text.