JuliaParallel / JuliaParallel/DTables.jl
How does one make DTable construction lazy?
Open
data movement
performance
table interface
upstream
- Dominant language
- Julia
- Stars
- 81
- Forks
- 2
- Avg merge
- 2h 31m
- Merged PRs (30d)
- 1
Description
I tried
tbl = Dagger.DTable(Parquet.read_parquet, my_files)
where "my_files" is an array of paths to parquet files that were from a dask dataframe. It seems to be loading everything into memory. I'd like a way to process out-of-core, similar to dask, I was under the impression this was a goal for DTable. Thanks.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.