lz4docstore batched storage
- Lingua principale
- Python
- Stelle
- 391
- Fork
- 58
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
**Is your feature request related to a problem? Please describe.**
Although the lz4docstore's bin file does a reasonably good job compressing full documents, resulting in a file size between gzip and uncompressed [1], it does much worse over collections with short passages and provides nearly no benefit over uncompressed source files and does much worse than gzip [2]. This is because it cannot benefit from inter-passage compression; within a single passage, there's just not very much redundancy.
[1] e.g., for `msmarco-document` docstore is 14G, gzip source is 8G, uncompressed source is 23GB
[2] e.g., for `msmarco-passage` docstore is 3G, gzip source is 1G, uncompressed source is 3G
**Describe the solution you'd like**
Update lz4docstore to support compressing multiple passages into the same batch. This will increase the context available for lz4 to compress (common pickle structure, common English words, etc.), while hopefully not having a huge effect on lookup times (if data stays within a page or two). Some collections also place passages from the same document nearby, further increasing the possible compression ratio.
When performing lookups, multiple doc_ids will point to the same position. It will need to decompress all and find the document the user requested.
When performing lookups by index, a new file will be needed to indicate which index within the chunk the index is at. If there's a maximum of 16 passages per chunk, there will be an additional storage overhead of 1/2 byte per document, or ~4.5MB for a dataset like `msmarco-passage`. Not so bad, considering the possible storage savings.
The dataset definition should indicate whether it uses chunks when creating the docstore. This value can be stored in the `bin.meta` file. This functionality should default to off.
This could probably be changed without migration; existing users will continue to use docstores without the inter-passage compression.
**Describe alternatives you've considered**
None considered.
**Additional context**
Known affected datasets:
- `msmarco-passage`
- `mr-tydi`
- `dpr-w100`
- probably `msmarco-qna`
- probably `natural-questions`
- probably `car/v1.5`
- probably some of the `beir` ones
- maybe `tripclick`
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.