queryverse / queryverse/Query.jl

@groupby failed on CategoricalString with missing value

Open
#278 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug upstream
Dominant language
Julia
Stars
403
Forks
48
Avg merge
3d 6h
Merged PRs (30d)
6

Description

Let's say, we try to @groupby on a CategoricalString column

df = DataFrame(
    fruit    =  ["Apple", "Banana", "Cherry", "Apple", "Banana", "Cherry"],
    price    =  [1.2, 2.0, 0.4, 1.2, 2.0, 0.4])

categorical!(df, :fruit)
df |> @groupby(_.fruit) |> @map({key=key(_), m=mean(_.price)})

The result is exactly as expected.

3×2 DataFrame
│ Row │ key          │ m       │
│     │ Categorical… │ Float64 │
├─────┼──────────────┼─────────┤
│ 1   │ Apple        │ 1.2     │
│ 2   │ Banana       │ 2.0     │
│ 3   │ Cherry       │ 0.4     │

Next, we make the same column contain some missing values.

df = DataFrame(
    fruit    =  ["Apple", missing, "Cherry", "Apple", "Banana", "Cherry"],
    price    =  [1.2, 2.0, 0.4, 1.2, 2.0, 0.4])

categorical!(df, :fruit)
df |> @groupby(_.fruit) |> @map({key=key(_), m=mean(_.price)})

Then, if failed with an error message stating somewhat like this.

ERROR: MethodError: convert(::Type{DataValue{CategoricalString{UInt32}}}, ::CategoricalString{UInt32}) is ambiguous

However, this behavior is weird since String column with similar missing values does not issue any complaints.

df = DataFrame(
    fruit    =  ["Apple", missing, "Cherry", "Apple", "Banana", "Cherry"],
    price    =  [1.2, 2.0, 0.4, 1.2, 2.0, 0.4])

df |> @groupby(_.fruit) |> @map({key=key(_), m=mean(_.price)})
key      │ m
─────────┼────
"Apple"  │ 1.2
#NA      │ 2.0
"Cherry" │ 0.4
"Banana" │ 2.0

This limitation may have to be fixed, since it is natural that String columns are recoded as Categorical columns before beginning any analysis.

Thanks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the two @groupby examples in the issue, comparing a CategoricalString column with missing values to the equivalent String column. Trace the @groupby handling that produces the ambiguous DataValue conversion. Done means the categorical example groups the missing value without the reported MethodError and preserves the expected results.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.