EleutherAI / EleutherAI/tokengrams
Add optional index sorting when MemmapIndex detects unsorted data
- Dominant language
- Rust
- Stars
- 28
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
# Feature Request: Add Optional Index Sorting When `MemmapIndex` Detects Unsorted Data
## Description
Currently, when `MemmapIndex` (or related index types) detects that the index is **not sorted**, it does not attempt to sort it automatically.
This can be a problem for large datasets, as sorting manually requires additional scripts and repeated I/O, which is extremely time-consuming for multi-terabyte corpora.
## Proposed Feature
- When loading a `.bin` or `.memmap` index, if unsorted order is detected, add an **optional flag** (e.g., `sort_if_unsorted=True`) to automatically sort the index before proceeding.
- Sorting should be done **in-place** if possible, or to a **temporary file** if memory constraints require.
- Log a **warning** when sorting is performed, including **time taken**.
Example:
```python
index = MemmapIndex.load(path, sort_if_unsorted=True)
```
## Motivation
- For large datasets (e.g., multiple TBs), discovering an unsorted index late in a processing pipeline is costly.
- A built-in sorting option would save users from having to rerun a separate preprocessing step.
- This is especially useful in workflows where indexes are generated externally or on distributed systems where ordering is not guaranteed.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.