JuliaParallel / JuliaParallel/DTables.jl

select, transform, combine not working on a DTable which is part of GDTable

Open
#68 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
81
Forks
2
Avg merge
2h 31m
Merged PRs (30d)
1

Description

because `select`, `transform`, and `combine` do not work on GDTables directly, the intuitive workaround is to apply them separately onto the different DTables which are part of GDTable.

This however fails because of some convert issue which seems to be an implementation detail.

```julia
using Distributed
# add two further julia processes which could run on other machines
addprocs(2, exeflags="--threads=2")
# Distributed.@everywhere execute code on all machines
@everywhere using Dagger
# Dagger uses both Threads and Machines as processes
Dagger.all_processors()

using DTables, DataFrames, CSV, Statistics

url = "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv"
files = [url, url, url, url, url]

d = DTable(DataFrame ∘ CSV.File ∘ download, files)
g = DTables.groupby(d, :species)

e = first(g)[2]
combine(fetch(e), :sepal_width => mean) # works
combine(e, :sepal_width => mean) # fails
```

the error is as below

```julia
julia> combine(e, :sepal_width => mean) # fails
ERROR: ThunkFailedException:
Root Exception Type: RemoteException
Root Exception:
On worker 2:
MethodError: Cannot `convert` an object of type Int64 to an object of type Tuple{SentinelArrays.IndexIterator{Vector{Float64}}, Tuple{Vector{Vector{Float64}}, Int64, Vector{Float64}, Int64, Int64, Int64}}

Closest candidates are:
convert(::Type{T}, ::T) where T<:Tuple
@ Base essentials.jl:456
convert(::Type{T}, ::T) where T
@ Base Base.jl:84
convert(::Type{T}, ::Tuple{Vararg{Any, N}}) where {N, T<:Tuple}
@ Base essentials.jl:457
...

Stacktrace:
[1] cvt1
@ ./essentials.jl:468 [inlined]
[2] ntuple
@ ./ntuple.jl:49 [inlined]
[3] convert
@ ./essentials.jl:470
[4] convert
@ ./some.jl:37
[5] setproperty!
@ ./Base.jl:40
[6] pull_next_chunk!
@ ~/.julia/packages/DTables/EiSy4/src/table/dtable_column.jl:53
[7] iterate
@ ~/.julia/packages/DTables/EiSy4/src/table/dtable_column.jl:67
[8] mean
@ ~/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/share/julia/stdlib/v1.10/Statistics/src/Statistics.jl:62
[9] mean
@ ~/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/share/julia/stdlib/v1.10/Statistics/src/Statistics.jl:44
[10] #invokelatest#2
@ ./essentials.jl:892 [inlined]
[11] invokelatest
@ ./essentials.jl:889 [inlined]
[12] #41
@ ~/.julia/packages/Dagger/Tx54v/src/threadproc.jl:20
Stacktrace:
[1] wait
@ ./task.jl:352 [inlined]
[2] fetch
@ ./task.jl:372 [inlined]
[3] #execute!#40
@ ~/.julia/packages/Dagger/Tx54v/src/threadproc.jl:26
[4] execute!
@ ~/.julia/packages/Dagger/Tx54v/src/threadproc.jl:13
[5] #167
@ ~/.julia/packages/Dagger/Tx54v/src/sch/Sch.jl:1611 [inlined]
[6] #21
@ ~/.julia/packages/Dagger/Tx54v/src/options.jl:18 [inlined]
[7] #1
@ ~/.julia/packages/ScopedValues/Kvcrb/src/ScopedValues.jl:163
[8] with_logstate
@ ./logging.jl:515
[9] with_logger
@ ./logging.jl:627 [inlined]
[10] enter_scope
@ ~/.julia/packages/ScopedValues/Kvcrb/src/payloadlogger.jl:17 [inlined]
[11] with
@ ~/.julia/packages/ScopedValues/Kvcrb/src/ScopedValues.jl:162
[12] with_options
@ ~/.julia/packages/Dagger/Tx54v/src/options.jl:17
[13] do_task
@ ~/.julia/packages/Dagger/Tx54v/src/sch/Sch.jl:1609
[14] #143
@ ~/.julia/packages/Dagger/Tx54v/src/sch/Sch.jl:1302
Root Thunk: Thunk(id=29, mean(DTableColumn{Float64, Tuple{Float64, Tuple{SentinelArrays.IndexIterator{Vector{Float64}}, Tuple{Vector{Vector{Float64}}, Int64, Vector{Float64}, Int64, Int64, Int64}}}}(DTable with 1 partitions
Tabletype: unknown (use `tabletype!(::DTable)`), 2, :sepal_width, [250], 0, nothing, nothing)))
Inner Thunk: Thunk(id=30, length(Thunk[29](mean, ...)))
This Thunk: Thunk(id=30, length(Thunk[29](mean, ...)))
Stacktrace:
[1] fetch(t::Dagger.ThunkFuture; proc::OSProc, raw::Bool)
@ Dagger ~/.julia/packages/Dagger/Tx54v/src/eager_thunk.jl:16
[2] fetch
@ ~/.julia/packages/Dagger/Tx54v/src/eager_thunk.jl:11 [inlined]
[3] #fetch#73
@ ~/.julia/packages/Dagger/Tx54v/src/eager_thunk.jl:58 [inlined]
[4] fetch
@ ~/.julia/packages/Dagger/Tx54v/src/eager_thunk.jl:54 [inlined]
[5] _manipulate(df::DTable, normalized_cs::Vector{Any}, copycols::Bool, keeprows::Bool)
@ DTables ~/.julia/packages/DTables/EiSy4/src/operations/dataframes_interface.jl:97
[6] manipulate(df::DTable, cs::Any; copycols::Bool, keeprows::Bool, renamecols::Bool)
@ DTables ~/.julia/packages/DTables/EiSy4/src/operations/dataframes_interface.jl:47
[7] combine(df::DTable, args::Any; renamecols::Bool, threads::Bool)
@ DTables ~/.julia/packages/DTables/EiSy4/src/operations/dataframes_interface.jl:190
[8] combine(df::DTable, args::Any)
@ DTables ~/.julia/packages/DTables/EiSy4/src/operations/dataframes_interface.jl:189
[9] top-level scope
@ REPL[15]:1
```

EDIT:
The `reduce` function works luckily, but is not that convenient, especially for mean.
```
julia> fetch(reduce(+, e, cols=[:sepal_width])).sepal_width / length(e)
2.974
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.