lz4docstore batched storage
- 主要語言
- Python
- 星號
- 391
- 分支
- 58
- PR 合併指標
- 30 天內沒有已合併 PR
描述
**Is your feature request related to a problem? Please describe.**
Although the lz4docstore's bin file does a reasonably good job compressing full documents, resulting in a file size between gzip and uncompressed [1], it does much worse over collections with short passages and provides nearly no benefit over uncompressed source files and does much worse than gzip [2]. This is because it cannot benefit from inter-passage compression; within a single passage, there's just not very much redundancy.
[1] e.g., for `msmarco-document` docstore is 14G, gzip source is 8G, uncompressed source is 23GB
[2] e.g., for `msmarco-passage` docstore is 3G, gzip source is 1G, uncompressed source is 3G
**Describe the solution you'd like**
Update lz4docstore to support compressing multiple passages into the same batch. This will increase the context available for lz4 to compress (common pickle structure, common English words, etc.), while hopefully not having a huge effect on lookup times (if data stays within a page or two). Some collections also place passages from the same document nearby, further increasing the possible compression ratio.
When performing lookups, multiple doc_ids will point to the same position. It will need to decompress all and find the document the user requested.
When performing lookups by index, a new file will be needed to indicate which index within the chunk the index is at. If there's a maximum of 16 passages per chunk, there will be an additional storage overhead of 1/2 byte per document, or ~4.5MB for a dataset like `msmarco-passage`. Not so bad, considering the possible storage savings.
The dataset definition should indicate whether it uses chunks when creating the docstore. This value can be stored in the `bin.meta` file. This functionality should default to off.
This could probably be changed without migration; existing users will continue to use docstores without the inter-passage compression.
**Describe alternatives you've considered**
None considered.
**Additional context**
Known affected datasets:
- `msmarco-passage`
- `mr-tydi`
- `dpr-w100`
- probably `msmarco-qna`
- probably `natural-questions`
- probably `car/v1.5`
- probably some of the `beir` ones
- maybe `tripclick`
貢獻指南
這個儲存庫沒有索引到貢獻指南
評估
這個 Issue 還沒有評估資料。