JuliaData / JuliaData/SplitApplyCombine.jl
Not sure how best to handle `missing`
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 154
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
I'm not actually sure where the right place on the stack is to fix this, because it seems to cut across several layers.
Here's an example - Say I want to get the mean of each group, ignoring missings. Notice that for the foo=2 group, both elements are missing.
using SplitApplyCombine
table = [(foo=1, bar=rand()),
(foo=2, bar=missing),
(foo=3, bar=rand()),
(foo=1, bar=missing),
(foo=2, bar=missing),
(foo=3, bar=rand())]
map(mean ∘ skipmissing, group(r->r.foo, r->r.bar, table))
This throws the error MethodError: no method matching zero(::Type{Any})
This is the result of a cascade of things, most of which seem pretty reasonable in isolation, which is why it's not clear (to me anyways) what the right fix is
meandoesn't know how to handle an empty arrayAny[]. I don't think there's anything more reasonable formeanto do heretabledoesn't have usful type information (see https://github.com/JuliaLang/julia/issues/31077)groupseems to set the type of the dictionary elements based on theeltypeoftable.
I'm not sure if there's a good resolution to this. Even if group built up the groups iteratively rather than pre-allocating, for a group with only missings it would end up with an Array{Missing}, which still doesn't help mean figure out what a reasonable answer is.
My current workaround is to re-inject the type information, but it took some digging to figure out what the actual problem was, and is not pretty:
map(c->mean(Vector{Float64}(collect(skipmissing(c)))), group(r->r.foo, r->r.bar, table))
Another workaround is setting the type of table explicitly:
table = NamedTuple{(:foo, :bar), Tuple{Int64, Union{Missing,Float64}}}[
(foo=1, bar=rand()),
...
But that gets pretty verbose.
Any thoughts as the the best way to handle this?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Julia example with SplitApplyCombine's group, skipmissing, and mean, then inspect how type information is lost across those operations. Review the linked JuliaLang issue 31077 and compare the two workarounds described. Done means agreeing on behavior for groups containing only missing values and documenting or testing that decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100