Improve training documentation user journey and sidebar naming
@prrao87 is already working on this.
Since Jul 22, 2026.
- Dominant language
- Python
- Stars
- 11
- Forks
- 11
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 14
Description
Clarify the journey between the training data-loading pages
After the changes in #294, training/index.mdx now provides a fairly complete introduction to loading training data with LanceDB. It covers direct table access, StreamingDataset, filtering, shuffling, transformations, distributed splits, and resumability. It ends by briefly introducing Permutation as the lower-level alternative.
The next sidebar page is currently called PyTorch integration. That title is broad, and the page initially appears to restart the data-loading discussion with Table, Permutation, and DataLoader. For someone reading the pages in sequence, it is not immediately clear why they have moved to a new page or how its purpose differs from the one they just finished.
The underlying distinction is useful and should remain:
Tableis the simplest direct-access option.StreamingDatasetis the complete streaming loader, with prefetching, transforms, distributed splitting, elastic determinism, and resumability.Permutationis the lower-level, map-style option for users who want explicit control over row selection and ordering while managing batching through PyTorch.
The documentation should make that choice clear. The first page should remain focused on the overall training-data workflow and the recommended StreamingDataset path. Its final section should explain when someone would choose Permutation instead and then point naturally to the PyTorch page.
The PyTorch page should be framed specifically around using Table and Permutation with torch.utils.data.DataLoader. Its unique value is the practical integration detail: collation, output formats, column selection, worker processes, remote connections, and connection factories. It should not read like a second general introduction to data loading.
A title such as PyTorch DataLoader would make this narrower purpose clearer than PyTorch integration. The page should also contain an explicit section such as Use a permutation with DataLoader, giving the preceding page a meaningful destination to link to.
This issue is about titles, scope, and narrative continuity between the two existing pages. It is not intended to move StreamingDataset into a separate document or substantially rewrite the feature documentation from #294.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.