lancedb / lancedb/docs

Improve training documentation user journey and sidebar naming

Open
#318 0 comments 0 reactions 1 assignee View on GitHub

@prrao87 is already working on this.

Since Jul 22, 2026.

Dominant language
Python
Stars
11
Forks
11
Avg merge
1d 13h
Merged PRs (30d)
14

Description

Clarify the journey between the training data-loading pages

After the changes in #294, training/index.mdx now provides a fairly complete introduction to loading training data with LanceDB. It covers direct table access, StreamingDataset, filtering, shuffling, transformations, distributed splits, and resumability. It ends by briefly introducing Permutation as the lower-level alternative.

The next sidebar page is currently called PyTorch integration. That title is broad, and the page initially appears to restart the data-loading discussion with Table, Permutation, and DataLoader. For someone reading the pages in sequence, it is not immediately clear why they have moved to a new page or how its purpose differs from the one they just finished.

The underlying distinction is useful and should remain:

  • Table is the simplest direct-access option.
  • StreamingDataset is the complete streaming loader, with prefetching, transforms, distributed splitting, elastic determinism, and resumability.
  • Permutation is the lower-level, map-style option for users who want explicit control over row selection and ordering while managing batching through PyTorch.

The documentation should make that choice clear. The first page should remain focused on the overall training-data workflow and the recommended StreamingDataset path. Its final section should explain when someone would choose Permutation instead and then point naturally to the PyTorch page.

The PyTorch page should be framed specifically around using Table and Permutation with torch.utils.data.DataLoader. Its unique value is the practical integration detail: collation, output formats, column selection, worker processes, remote connections, and connection factories. It should not read like a second general introduction to data loading.

A title such as PyTorch DataLoader would make this narrower purpose clearer than PyTorch integration. The page should also contain an explicit section such as Use a permutation with DataLoader, giving the preceding page a meaningful destination to link to.

This issue is about titles, scope, and narrative continuity between the two existing pages. It is not intended to move StreamingDataset into a separate document or substantially rewrite the feature documentation from #294.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.