apache / apache/datafusion

"Cannot infer common argument type for comparison operation Union..."

Open
#10,180 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

See https://github.com/datafusion-contrib/datafusion-functions-json/pull/3

I have a union defined by

```rs
DataType::Union(
UnionFields::new(
vec![0, 1, 2, 3, 4, 5, 6],
vec![
Field::new("null", DataType::Boolean, true),
Field::new("bool", DataType::Boolean, false),
Field::new("int", DataType::Int64, false),
Field::new("float", DataType::Float64, false),
Field::new("string", DataType::Utf8, false),
Field::new("array", DataType::Utf8, false),
Field::new("object", DataType::Utf8, false),
]
),
UnionMode::Sparse,
)
```

When I try to compare it to an integer with `json_get(json_data, 'foo')=123`, I get the error:

```
called `Result::unwrap()` on an `Err` value: Plan("Cannot infer common argument type for comparison operation Union([(0, Field { name: \"null\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), (1, Field { name: \"bool\", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }), (2, Field { name: \"int\", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }), (3, Field { name: \"float\", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }), (4, Field { name: \"string\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }), (5, Field { name: \"array\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }), (6, Field { name: \"object\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} })], Sparse) = Int64")
```

Is there any way to rewire the logic plan to tell datafusion how to implement these comparisons?

If not, I might have to abandon the `json_get` method, and instead implemtn `json_get_str`, `json_get_int` etc., which would be unfortunate.

I tried implementing `FunctionRewrite`, but the error occurs before it's called.

### To Reproduce

see tests in https://github.com/datafusion-contrib/datafusion-functions-json/pull/3

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the tests referenced in datafusion-functions-json/pull/3 and trace comparison type inference before FunctionRewrite, since the reported error occurs first. Confirm the intended behavior for json_get(json_data, 'foo')=123 and add a regression test that covers the Union-versus-Int64 comparison.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.