CommonCrawl Indexing Code

Is your feature request related to a problem? Please describe.

We are looking to index a sizeable fraction of CommonCrawl: a few hundred million domains from a whitelist.

Describe the solution you'd like

The CommonCrawl demo and the Mission of lowering the costs of indexing the web to $1K/month have been core to Quickwit from the beginning.
https://common-crawl.quickwit.io/

I've been looking for a while for the implementation of the demo, but I can't find it anywhere. I'm particularly looking for the code for preprocessing and indexing the CommonCrawl dataset as described in the blogpost.

Can you point to it if it is public? Or could you consider open-sourcing it?

Describe alternatives you've considered

It wouldn't be too challenging to parse CommonCrawl and index into Quickwit with some parallelization for speed, but it would be helpful to have your well-tuned implementation since it already exists.

The issue mentions the CommonCrawl demo and its blog post but names no repository files, tests, or entry points. Start by locating the demo and the preprocessing/indexing implementation described there. Done would be either identifying the existing public code or defining and implementing an open-source version, which is not scoped here.