Implementation of line-wise corrections

Hi,

Thanks for your efforts in open-sourcing such an awesome library for large-scale data processing.

I'm trying to reproduce the RefinedWeb dataset. I find that the pipeline for processing common crawl is very similar to the MDR pipeline used in RefinedWeb. But I couldn't find the part of line-wise corrections in the code.


Could you add more details on this?

Thanks