queryverse / queryverse/Query.jl

@mutate changes other column type

Open
#305 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
403
Forks
48
Avg merge
3d 6h
Merged PRs (30d)
6

Description

Repost from https://discourse.julialang.org/t/query-mutate-changes-other-column-type/39292 as it looks like a bug.

This is my smallest code with repro:

contents = """
"5674012","aa66aa66"
"5674012","b036aa66,b036aa67,b036aa68"
""";

batches = CSV.File(IOBuffer(contents); header = ["X1", "Splits"], delim = ',') |> DataFrame;

emptyStringArray = Array{SubString{String},1}()
batchesAny = batches |> 
            @mutate(Splits = length(_.Splits) > 0 ? split(_.Splits, ',') : emptyStringArray) |> DataFrame
batchesString = batches |> 
            @mutate(Splits = length(_.Splits) > 0 ? split(_.Splits, ',') : Array{SubString{String},1}()) |> DataFrame

The output is like this:

julia> batchesAny
2×2 DataFrame
│ Row │ X1      │ Splits                               │
│     │ Any     │ Any                                  │
├─────┼─────────┼──────────────────────────────────────┤
│ 1   │ 5674012 │ ["aa66aa66"]                         │
│ 2   │ 5674012 │ ["b036aa66", "b036aa67", "b036aa68"] │

julia> batchesString
2×2 DataFrame
│ Row │ X1      │ Splits                               │
│     │ Int64   │ Array{SubString{String},1}           │
├─────┼─────────┼──────────────────────────────────────┤
│ 1   │ 5674012 │ ["aa66aa66"]                         │
│ 2   │ 5674012 │ ["b036aa66", "b036aa67", "b036aa68"] │

What I don’t understand:

  1. :Splits column type differs - Any vs. Array{SubString{String},1}. (I just wanted to save memory so I stored the value (that can be repeated) to emptyStringArray.)

  2. Even if I understand that I made something bad to column :Splits, I think X1's type shouldn't be changed to Any in batchesAny.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the Julia reproduction in the issue and inspect the implementation of @mutate and its DataFrame conversion path. Compare batchesAny with batchesString and verify that changing Splits does not also change X1 from Int64 to Any; done means the regression is covered and the demonstrated output preserves X1's type.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.