queryverse / queryverse/Query.jl
@groupby failed on CategoricalString with missing value
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 403
- Forks
- 48
- Avg merge
- 3d 6h
- Merged PRs (30d)
- 6
Description
Let's say, we try to @groupby on a CategoricalString column
df = DataFrame(
fruit = ["Apple", "Banana", "Cherry", "Apple", "Banana", "Cherry"],
price = [1.2, 2.0, 0.4, 1.2, 2.0, 0.4])
categorical!(df, :fruit)
df |> @groupby(_.fruit) |> @map({key=key(_), m=mean(_.price)})
The result is exactly as expected.
3×2 DataFrame
│ Row │ key │ m │
│ │ Categorical… │ Float64 │
├─────┼──────────────┼─────────┤
│ 1 │ Apple │ 1.2 │
│ 2 │ Banana │ 2.0 │
│ 3 │ Cherry │ 0.4 │
Next, we make the same column contain some missing values.
df = DataFrame(
fruit = ["Apple", missing, "Cherry", "Apple", "Banana", "Cherry"],
price = [1.2, 2.0, 0.4, 1.2, 2.0, 0.4])
categorical!(df, :fruit)
df |> @groupby(_.fruit) |> @map({key=key(_), m=mean(_.price)})
Then, if failed with an error message stating somewhat like this.
ERROR: MethodError: convert(::Type{DataValue{CategoricalString{UInt32}}}, ::CategoricalString{UInt32}) is ambiguous
However, this behavior is weird since String column with similar missing values does not issue any complaints.
df = DataFrame(
fruit = ["Apple", missing, "Cherry", "Apple", "Banana", "Cherry"],
price = [1.2, 2.0, 0.4, 1.2, 2.0, 0.4])
df |> @groupby(_.fruit) |> @map({key=key(_), m=mean(_.price)})
key │ m
─────────┼────
"Apple" │ 1.2
#NA │ 2.0
"Cherry" │ 0.4
"Banana" │ 2.0
This limitation may have to be fixed, since it is natural that String columns are recoded as Categorical columns before beginning any analysis.
Thanks.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the two @groupby examples in the issue, comparing a CategoricalString column with missing values to the equivalent String column. Trace the @groupby handling that produces the ambiguous DataValue conversion. Done means the categorical example groups the missing value without the reported MethodError and preserves the expected results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100