JuliaParallel / JuliaParallel/DTables.jl

Let `mapreduce` support `GDTable`

Open
#69 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
81
Forks
2
Avg merge
2h 31m
Merged PRs (30d)
1

Description

`map` and `reduce` all support `GDTable`, but `mapreduce` does not.

```julia
using Distributed
# add two further julia processes which could run on other machines
addprocs(2, exeflags="--threads=2")
# Distributed.@everywhere execute code on all machines
@everywhere using Dagger
# Dagger uses both Threads and Machines as processes
Dagger.all_processors()

using DTables, DataFrames, CSV, OnlineStats

url = "[https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv"](https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv%22)
files = [url, url, url, url, url]

d = DTable(DataFrame ∘ CSV.File ∘ download, files)
g = DTables.groupby(d, :species)

fetch(reduce(fit!, map(r -> (r.sepal_width,), g), init=Mean())) # works
fetch(mapreduce(r -> (r.sepal_width,), fit!, g, init= Mean())) # fails
```

the error is below
```julia
julia> fetch(mapreduce(r -> (r.sepal_width,), fit!, g, init= Mean()))
ERROR: type Pair has no field sepal_width
Stacktrace:
[1] getproperty
@ ./Base.jl:37 [inlined]
[2] (::var"#23#24")(r::Pair{String15, DTable})
@ Main ./REPL[44]:1
[3] MappingRF
@ ./reduce.jl:100 [inlined]
[4] _foldl_impl
@ ./reduce.jl:58 [inlined]
[5] foldl_impl
@ ./reduce.jl:48 [inlined]
[6] mapfoldl_impl(f::var"#23#24", op::typeof(fit!), nt::Mean{Float64, EqualWeight}, itr::DTables.GDTable)
@ Base ./reduce.jl:44
[7] mapfoldl(f::Function, op::Function, itr::DTables.GDTable; init::Mean{Float64, EqualWeight})
@ Base ./reduce.jl:175
[8] mapfoldl
@ ./reduce.jl:175 [inlined]
[9] #mapreduce#302
@ ./reduce.jl:307 [inlined]
[10] top-level scope
@ REPL[44]:1
```

As the documentation recommends to use `mapreduce` over `map` + `reduce`, it would be really good if mapreduce also supports GDTables.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.