Saving processed dataset running infinitely

I have a text dataset of size 220M.

For pre-processing, I need to tokenize this and filter rows with the large sequence.

My tokenization took roughly 3hrs. I used map() with batch size 1024 and multi-process with 96 processes.

filter() function was way to slow, so I used a hack to use pyarrow filter table function, which is damm fast. Mentioned [here](https://github.com/huggingface/datasets/issues/1796)

```dataset._data = dataset._data.filter(...)```
It took 1 hr for the filter.

Then i use `save_to_disk()` on processed dataset and it is running forever.

I have been waiting since 8 hrs, it has not written a single byte.

Infact it has actually read from disk more than 100GB, screenshot below shows the stats using `iotop`.
Second process is the one.

I am not able to figure out, whether this is some issue with dataset library or that it is due to my hack for filter() function.