google-research / google-research/dex-lang

Create a Dex datasets library

Open
#458 4 comments 0 reactions 0 assignees View on GitHub
libraries
Dominant language
Haskell
Stars
1.7k
Forks
116
PR merge metrics
No merged PRs in 30d

Description

## Motivation

Create a structured datasets library within Dex: `lib/datasets.dx`.

The library should enable straightforward usage of machine learning datasets, including the following:
- [ ] Downloading datasets to a shared machine location (in `/tmp` or `~/.dex/datasets/...`)
- [ ] Unzipping datasets, handling various compression formats.
- [ ] Parsing: loading the dataset as a Dex data structure.
- Example: input-output pairs. `List (inputSize => Float & labelSize => Int)`
- [ ] Transforms: data transformations - batching, shuffling (nondeterminism), concatenation, filtering, mapping, and augmentation.

## Implementation ideas

- Dataset downloading and unzipping could be implemented via shell command support in Dex using `IO` effect.
- Example: `wget` a named dataset with library-hardcoded URL to `~/.dex/datasets/...` if it doesn't already exist.
- Parsing could be implemented using parser combinators, or ad-hoc string processing logic.
- Transforms could be implemented using `Accum` effect for MapReduce-like functionality and potential for parallelism.

## Prior work

- [TensorFlow Datasets](https://github.com/tensorflow/datasets): downloading & unzipping & transforms
- [Swift for TensorFlow model support](https://github.com/tensorflow/swift-models/blob/243bfb4a2344ab578bc905f5db6d547e07fdc651/Support/FileManagement.swift): downloading & unzipping
- [`createDirectoryIfMissing(at:)`](https://github.com/tensorflow/swift-models/blob/243bfb4a2344ab578bc905f5db6d547e07fdc651/Support/FileManagement.swift#L25), [`download(from:to:)`](https://github.com/tensorflow/swift-models/blob/243bfb4a2344ab578bc905f5db6d547e07fdc651/Support/FileManagement.swift#L43), [`extractArchive(at:to:fileExtension:deleteArchiveWhenDone:)`](https://github.com/tensorflow/swift-models/blob/243bfb4a2344ab578bc905f5db6d547e07fdc651/Support/FileManagement.swift#L100)
- [oxinabox/DataDeps.jl](https://github.com/oxinabox/DataDeps.jl): downloading & unzipping
- [JuliaML/MLDatasets.jl](https://github.com/JuliaML/MLDatasets.jl): parsing & download/unzip based on DataDeps.jl
- [JuliaText/CorpusLoaders.jl](https://github.com/JuliaText/CorpusLoaders.jl): parsing & download/unzip based on DataDeps.jl
- [NLTK datasets](https://www.nltk.org/book/ch02.html): downloading & unzipping & parsing

Contributor guide

Open the contributing guide

Research direction

Start with the proposed entry point, lib/datasets.dx, and review the issue's prior-work references for downloading, archive extraction, parsing, and transformations. The issue describes the intended capabilities but does not name tests or a narrower first milestone; done would require implementing the full structured datasets library and its listed operations.

Written by the indexing model from the issue text.

Assessment

Domain
data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.