JuliaML / JuliaML/MLDatasets.jl

write datasets in a JLD2 or Arrow format for faster read

Open
#125 6 comments 0 reactions 0 assignees View on GitHub
gsoc
Dominant language
Julia
Stars
239
Forks
51
PR merge metrics
No merged PRs in 30d

Description

We could have a "processed" folder in each dataset folder where we write the dataset object the first time we create it. In the following creations, e.g. `d = MNIST()` we just load the JLD2 file.

Example:
```julia
function MNIST(...)
dataset_dir = ...
processed_file = joinpath(dataset_dir, "processed", "dataset.jld2")
if isfile(processed_file)
return FileIO.load(processed_file, "dataset")
end

mnist = ...
if isfile(processed_file)
FileIO.save(processed_file, Dict("dataset" => mnist))
end
return mnist
end
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.