JuliaML / JuliaML/MLDatasets.jl
write datasets in a JLD2 or Arrow format for faster read
- Dominant language
- Julia
- Stars
- 239
- Forks
- 51
- PR merge metrics
- No merged PRs in 30d
Description
We could have a "processed" folder in each dataset folder where we write the dataset object the first time we create it. In the following creations, e.g. `d = MNIST()` we just load the JLD2 file.
Example:
```julia
function MNIST(...)
dataset_dir = ...
processed_file = joinpath(dataset_dir, "processed", "dataset.jld2")
if isfile(processed_file)
return FileIO.load(processed_file, "dataset")
end
mnist = ...
if isfile(processed_file)
FileIO.save(processed_file, Dict("dataset" => mnist))
end
return mnist
end
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.