Unexpected performance degradation behavior in minhash deduplication stage 2

I've been running some large-scale benchmarking with minhash deduplication on SLURM clusters, loosely following [this example](https://github.com/huggingface/datatrove/blob/main/examples/minhash_deduplication.py)

The benchmarks consist of running stages 1 and 2 with the following configurations:
* Two input datasets: 4TB and 8TB
* Number of tasks for stage 1: 1200 and 2400
* Number of tasks for stage 2: 300 and 600 (also tried 1200 once, see below)

What I'm observing is that `stage 1` seems to scale fairly linearly between these configs. I have the following timing values in the final stats file (**all values are in minutes**):

| Stage 1 tasks vs. Dataset size | 4TB | 8TB |
| ---------- | --- | --- |
| **1200 tasks** | 71 minutes | 123 minutes |
| **2400 tasks** | 36 minutes | 61 minutes |

However, for `stage 2`, the scaling becomes quite different, especially when running the 8TB configuration:

| Stage 2 tasks vs. Dataset size | 4TB (stage 1 done with 1200 tasks) | 8TB (stage 1 done with 2400 tasks) |
| --------- | ------------------------------------------ | ------------------------------------------ |
| **300 tasks** | 25 minutes | 38 minutes |
| **600 tasks** | 17 minutes | **> 54 minutes <** |
| **1200 tasks** | - | **> 50 minutes <** |

As reported above, the some 8TB configs for `stage 2` (boldfaced) is taking an unexpectedly long time to run. I repeated these experiments several times, and the results appear consistent.

I was wondering if this behavior is expected? If so, what could be a possible explanation?
Let me know if I can provide further information.

Start with examples/minhash_deduplication.py and the stage 1 and stage 2 configurations described in the report. Compare the repeated 4TB and 8TB benchmark results, focusing on why increasing stage 2 tasks correlates with longer 8TB runs. Done means determining whether the behavior is expected and documenting a supported explanation.

Written by the indexing model from the issue text.