EleutherAI / EleutherAI/tokengrams

Add optional index sorting when MemmapIndex detects unsorted data

Open
#24 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
28
Forks
3
PR merge metrics
No merged PRs in 30d

Description

# Feature Request: Add Optional Index Sorting When `MemmapIndex` Detects Unsorted Data

## Description
Currently, when `MemmapIndex` (or related index types) detects that the index is **not sorted**, it does not attempt to sort it automatically.
This can be a problem for large datasets, as sorting manually requires additional scripts and repeated I/O, which is extremely time-consuming for multi-terabyte corpora.

## Proposed Feature
- When loading a `.bin` or `.memmap` index, if unsorted order is detected, add an **optional flag** (e.g., `sort_if_unsorted=True`) to automatically sort the index before proceeding.
- Sorting should be done **in-place** if possible, or to a **temporary file** if memory constraints require.
- Log a **warning** when sorting is performed, including **time taken**.

Example:
```python
index = MemmapIndex.load(path, sort_if_unsorted=True)
```

## Motivation
- For large datasets (e.g., multiple TBs), discovering an unsorted index late in a processing pipeline is costly.
- A built-in sorting option would save users from having to rerun a separate preprocessing step.
- This is especially useful in workflows where indexes are generated externally or on distributed systems where ordering is not guaranteed.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.