JuliaData / JuliaData/SplitApplyCombine.jl

Not sure how best to handle `missing`

Open
#22 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
154
Forks
16
PR merge metrics
No merged PRs in 30d

Description

I'm not actually sure where the right place on the stack is to fix this, because it seems to cut across several layers.

Here's an example - Say I want to get the mean of each group, ignoring missings. Notice that for the foo=2 group, both elements are missing.

using SplitApplyCombine

table = [(foo=1, bar=rand()),
         (foo=2, bar=missing),
         (foo=3, bar=rand()),
         (foo=1, bar=missing),
         (foo=2, bar=missing),
         (foo=3, bar=rand())]
map(mean ∘ skipmissing, group(r->r.foo, r->r.bar, table))

This throws the error MethodError: no method matching zero(::Type{Any})

This is the result of a cascade of things, most of which seem pretty reasonable in isolation, which is why it's not clear (to me anyways) what the right fix is

  1. mean doesn't know how to handle an empty array Any[]. I don't think there's anything more reasonable for mean to do here
  2. table doesn't have usful type information (see https://github.com/JuliaLang/julia/issues/31077)
  3. group seems to set the type of the dictionary elements based on the eltype of table.

I'm not sure if there's a good resolution to this. Even if group built up the groups iteratively rather than pre-allocating, for a group with only missings it would end up with an Array{Missing}, which still doesn't help mean figure out what a reasonable answer is.

My current workaround is to re-inject the type information, but it took some digging to figure out what the actual problem was, and is not pretty:

map(c->mean(Vector{Float64}(collect(skipmissing(c)))), group(r->r.foo, r->r.bar, table))

Another workaround is setting the type of table explicitly:

table = NamedTuple{(:foo, :bar), Tuple{Int64, Union{Missing,Float64}}}[
    (foo=1, bar=rand()),
    ...

But that gets pretty verbose.

Any thoughts as the the best way to handle this?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Julia example with SplitApplyCombine's group, skipmissing, and mean, then inspect how type information is lost across those operations. Review the linked JuliaLang issue 31077 and compare the two workarounds described. Done means agreeing on behavior for groups containing only missing values and documenting or testing that decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.