google/grain

View on GitHub

Library for reading and processing ML training data.

Stars: 779
Forks: 86
Open beginner issues: 0
Indexed issues: 41
Avg merge: 2d 6h
Merged PRs (30d): 6

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 15, 2026
Latest indexed: Sep 16, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Open All

41 open issues indexed

error Open

type:support

google/grain#333 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Slow data loading of large arrays from sharded dataset Open

type:feature

google/grain#372 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Simple usage example? (eg. MNIST) Open

type:support

google/grain#381 · 6 comments · 13 reactions · 0 assignees · Indexed Sep 16, 2026
AttributeError: 'NoneType' object has no attribute 'mmap' Open

type:bug

google/grain#398 · 4 comments · 6 reactions · 0 assignees · Indexed Sep 16, 2026
whether to support streaming dataset？ Open

type:feature

google/grain#487 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Batching into shared memory is deprecated, but essential for performance Open

type:performance

google/grain#492 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Advice on using a JIT function inside a __getitem__ or transform? Open

type:performance

google/grain#495 · 3 comments · 1 reaction · 0 assignees · Indexed Sep 16, 2026
Slow compared to simple loader Open

type:performance

google/grain#569 · 5 comments · 1 reaction · 0 assignees · Indexed Sep 16, 2026
Nested Dataset Slowdown Open

type:performance

google/grain#593 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
How would one create a data source for video? Open

type:support

google/grain#633 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Build from source succeeds but subsequent `import`s fail - also ERROR: Configuration file not found: grain/third_party/py/grain/oss/copy.bara.sky Open

type:bug

google/grain#669 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
comparing array_record's ArrayRecordDataSource + grain data loader to huggingface. (Grain is extremely slow) Open

type:performance

google/grain#688 · 0 comments · 3 reactions · 0 assignees · Indexed Sep 16, 2026
error: could not create 'build/lib.macosx-11.0-arm64-cpython-312/grain': Not a directory Open

type:build/install

google/grain#694 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Correct way to consume multiple files Open

type:support

google/grain#708 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Windows support Open

type:feature

google/grain#793 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Amount of time spent accessing thread locks for ArrayRecord datasources Open

type:performance

google/grain#808 · 1 comment · 1 reaction · 0 assignees · Indexed Sep 16, 2026
osx cant open .so array_ Open

type:bug

google/grain#812 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Grain workers allocates GPU memory Open

type:performance

google/grain#820 · 1 comment · 2 reactions · 1 assignee · Indexed Sep 16, 2026
SharedMemoryArray is too slow Open

type:performance

google/grain#851 · 3 comments · 2 reactions · 0 assignees · Indexed Sep 16, 2026
Discussion: Grain packaging nightly builds Open

type:build/install

google/grain#878 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Feature: Define transforms via Protocol instead of abstract class Open

type:feature

google/grain#884 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Document backwards compatibility guarantees Open

type:feature

google/grain#898 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 16, 2026
DataLoader ignores context manager for Data Sources Open

type:support

google/grain#936 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Add dataloader checkpointing example to docs Open

documentation type:feature

google/grain#938 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
`MapDataset` does not respect `jax.default_device(jax.devices("cpu")[0])`. Open

type:bug

google/grain#975 · 3 comments · 1 reaction · 0 assignees · Indexed Sep 16, 2026
`ConcatThenSplitIterDataset` producing incorrect results Open

type:bug

google/grain#981 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Mutex lock conflict between datasets and grain libraries with array_record>=0.8 Open

type:bug

google/grain#1023 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Leaked shared_memory objects to clean up at shutdown Open

type:bug

google/grain#1031 · 1 comment · 2 reactions · 0 assignees · Indexed Sep 16, 2026
Cannot use grain with pyarrow on Mac M1 Pro Open

type:bug

google/grain#1045 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Are individual contributions being reviewed? Open

type:support

google/grain#1063 · 1 comment · 1 reaction · 0 assignees · Indexed Sep 16, 2026
grain dataset slow with huggingface data source Open

type:performance

google/grain#1084 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
random and augmentation in datasets Open

type:support

google/grain#1102 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Performance gap with `tf.data.Dataset` on MNIST Open

type:performance

google/grain#1164 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
`ThreadPrefetchIterDataset` memory leaks / zombie threads that hang exiting? Open

type:bug

google/grain#1196 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Grain workers use GPUs, and there is no way to avoid it Open

type:performance

google/grain#1199 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Add Windows Build to `conda-forge` Open

type:feature

google/grain#1232 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Issue on page /grain.experimental.html Open

documentation

google/grain#1238 · 4 comments · 1 reaction · 0 assignees · Indexed Sep 16, 2026
No support for python 3.13t and 3.14t Open

type:feature

google/grain#1245 · 1 comment · 3 reactions · 0 assignees · Indexed Sep 16, 2026
Sliding window operation in `IterDataset` / build grain locally Open

type:support

google/grain#1264 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 16, 2026
How are config options meant to be set in Grain? Open

type:support

google/grain#1360 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
ThreadPrefetch `close()` cannot cancel a producer blocked in `parent.__next__()` Open

type:bug

google/grain#1372 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026

google/grain

Get new issues in your inbox