Scalable merge/compaction of big doc values segments.

The question is regarding the scalable merge/compaction of doc values, given the following context:

* I have a large sharded index.
* Each shard can contain segments of millions of documents.
* There are several hundreds fields in the index, and half of them are doc values.

Sometimes, I face issues with merge times when I need to merge or compact a large segment. The problem is that it's a single-threaded operation where a single segment is merged in a single merger thread.

From the codec doc values format of version 9.x, it appears possible to use map-reduce techniques when writing new large doc value segments. This is because all metadata is read before any field data can be read, and all doc value types have offset and length fields in the metadata.

My basic idea is to write each field in parallel to a separate file and then perform a low-level merge of the binary data (just appending bytes to the final file). After that, I can rewrite only the metadata to update the offsets.

As I am still new to Lucene development, could someone please provide some critique of this idea?

p.s. Unfortunately, the same idea is not applicable for the inverted index format due to its complexity.

Start by reading the version 9.x codec doc values format and the existing segment merge/compaction path. Evaluate whether per-field parallel writes and a later binary merge can preserve the metadata offsets and lengths; done would require a concrete design and evidence that it improves large-segment merge times.

Written by the indexing model from the issue text.