google-research / google-research/dex-lang
Create a Dex datasets library
- Dominant language
- Haskell
- Stars
- 1.7k
- Forks
- 116
- PR merge metrics
- No merged PRs in 30d
Description
## Motivation
Create a structured datasets library within Dex: `lib/datasets.dx`.
The library should enable straightforward usage of machine learning datasets, including the following:
- [ ] Downloading datasets to a shared machine location (in `/tmp` or `~/.dex/datasets/...`)
- [ ] Unzipping datasets, handling various compression formats.
- [ ] Parsing: loading the dataset as a Dex data structure.
- Example: input-output pairs. `List (inputSize => Float & labelSize => Int)`
- [ ] Transforms: data transformations - batching, shuffling (nondeterminism), concatenation, filtering, mapping, and augmentation.
## Implementation ideas
- Dataset downloading and unzipping could be implemented via shell command support in Dex using `IO` effect.
- Example: `wget` a named dataset with library-hardcoded URL to `~/.dex/datasets/...` if it doesn't already exist.
- Parsing could be implemented using parser combinators, or ad-hoc string processing logic.
- Transforms could be implemented using `Accum` effect for MapReduce-like functionality and potential for parallelism.
## Prior work
- [TensorFlow Datasets](https://github.com/tensorflow/datasets): downloading & unzipping & transforms
- [Swift for TensorFlow model support](https://github.com/tensorflow/swift-models/blob/243bfb4a2344ab578bc905f5db6d547e07fdc651/Support/FileManagement.swift): downloading & unzipping
- [`createDirectoryIfMissing(at:)`](https://github.com/tensorflow/swift-models/blob/243bfb4a2344ab578bc905f5db6d547e07fdc651/Support/FileManagement.swift#L25), [`download(from:to:)`](https://github.com/tensorflow/swift-models/blob/243bfb4a2344ab578bc905f5db6d547e07fdc651/Support/FileManagement.swift#L43), [`extractArchive(at:to:fileExtension:deleteArchiveWhenDone:)`](https://github.com/tensorflow/swift-models/blob/243bfb4a2344ab578bc905f5db6d547e07fdc651/Support/FileManagement.swift#L100)
- [oxinabox/DataDeps.jl](https://github.com/oxinabox/DataDeps.jl): downloading & unzipping
- [JuliaML/MLDatasets.jl](https://github.com/JuliaML/MLDatasets.jl): parsing & download/unzip based on DataDeps.jl
- [JuliaText/CorpusLoaders.jl](https://github.com/JuliaText/CorpusLoaders.jl): parsing & download/unzip based on DataDeps.jl
- [NLTK datasets](https://www.nltk.org/book/ch02.html): downloading & unzipping & parsing
Contributor guide
Research direction
Start with the proposed entry point, lib/datasets.dx, and review the issue's prior-work references for downloading, archive extraction, parsing, and transformations. The issue describes the intended capabilities but does not name tests or a narrower first milestone; done would require implementing the full structured datasets library and its listed operations.
Written by the indexing model from the issue text.
Assessment
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100