ClueWeb22

**Dataset Information:**

ClueWeb22 is the newest in the Lemur Project's ClueWeb line of datasets that support research on information retrieval, natural language processing and related human language technologies. This new dataset is being developed by the Lemur Project with significant assistance and support from Microsoft Corporation.

The ClueWeb22 dataset has several novel characteristics compared with earlier ClueWeb datasets.
- It is much larger.
- Documents are of higher quality.
- Documents are provided in several formats (HTML, clean text, screen shots).
- Document page analyses are provided that reveal where on a page text was displayed, and what was near it.
- The dataset includes a large set of crowdsourced queries and shallow relevance assessments (a pseudo search log).

Authors: Arnold Overwijk, Chenyan Xiong (@xiongchenyan), Jamie Callan (@jamiecallan), Cameron VandenBerg, Xiao Lucy Liu

**Links to Resources:**

- Website: https://lemurproject.org/clueweb22/index.php
- Documents spec: https://lemurproject.org/clueweb22/docspecs.php
- Queries spec: https://lemurproject.org/clueweb22/qryspecs.php
- SIRIP paper: https://doi.org/10.1145/3477495.3536321

**Dataset ID(s) & supported entities:**

- `clueweb22/a`: 200M docs, queries, qrels, scoreddocs?
- `clueweb22/b`: 2B docs, queries?, qrels?, scoreddocs?
- `clueweb22/l`: 10B docs, queries?, qrels?, scoreddocs?

**Checklist**

Mark each task once completed. All should be checked prior to merging a new dataset.

- [x] Dataset definition (in `ir_datasets/datasets/clueweb22.py`)
- [x] Tests (in `tests/integration/clueweb22.py`)
- [x] Metadata generated (using `ir_datasets generate_metadata` command, should appear in `ir_datasets/etc/metadata.json`)
- [x] Documentation (in `ir_datasets/etc/clueweb22.yaml`)
- [ ] Documentation generated in https://github.com/seanmacavaney/ir-datasets.com/
- [x] Downloadable content (in `ir_datasets/etc/downloads.json`) _Manual download requirded._
- [x] Download instructions added
- [x] ~~Download verification action (in `.github/workflows/verify_downloads.yml`). Only one needed per `topid`.~~
- [x] ~~Any small public files from NIST (or other potentially troublesome files) mirrored in https://github.com/seanmacavaney/irds-mirror/. Mirrored status properly reflected in `downloads.json`.~~

**Additional comments/concerns/ideas/etc.**

The dataset is planned to be used for shared tasks in the near future.
I also personally think it is of very high value to have this in ir_datasets.

**Open Questions**
- [x] Where to get the topic tag mentioned in the [paper](https://doi.org/10.1145/3477495.3536321)?
- [x] Is `VDOM-Paragraph` the same as `VDOM-Passage` in the [WARC headers](https://lemurproject.org/clueweb22/docspecs.php)?
- [x] What means the `?` in the inlink format anchor [type description](https://lemurproject.org/clueweb22/docspecs.php)?