Add additional write stage for merging multiple Megatron `IndexedDataset` file pairs into one

Is your feature request related to a problem? Please describe.

Currently, when I use the MegatronTokenizerWriter, it creates individual .bin/.idx file pairs, for each worker. This is obviously needed for efficiency, but when running on a machine with a lot of vCPUs, I got >100 different file pairs - which make it difficult to work with.

Describe the solution you'd like

It would be very helpful to be able to add some post-processing stage which merges all of them into a single output file. It can either be part of the same stage, or an additional optional stage in the pipeline.

Describe alternatives you've considered

A workaround solution is to run the merge_datasets.py script from Megatron-LM. Ideally if this could be part of the pipeline, it would simplify the process - especially since installing megatron-core requires a lot of dependencies, and it would be simpler to work within a single env with NeMo-Curator.

Start with the MegatronTokenizerWriter and review the referenced Megatron-LM merge_datasets.py script to understand the expected merge behavior. Add an optional post-processing stage that combines the worker-generated .bin/.idx pairs into one output pair, then verify that it works within a NeMo-Curator pipeline without requiring megatron-core.