apache / apache/datafusion

Add tests for support for `DataType::Union`

Open
#10,161 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

As we saw on https://github.com/apache/datafusion/pull/10139 with @joroKr21 there is not any particularly good way to test the coverage of [`DataType::Union`](https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html#variant.Union) today with our tests.

Note this is different than the SQL `UNION` / `UNION ALL` feature in [union.slt](https://github.com/apache/datafusion/blob/16e3831734358d2c628c7ff281cddd680dc4aa10/datafusion/sqllogictest/test_files/union.slt)

### Describe the solution you'd like

I would like some way to test `DataType::Union` , ideally via sqllogictest -- in `union_type.slt`

### Describe alternatives you've considered

I think the ideal alternative is to create a function that creates a `UnionArray` from different columns and use that function to create some simple queries / data in `union_type.slt`

Something like `make_union(col1, col2, ...)` (similarly to [`make_array`](https://arrow.apache.org/datafusion/user-guide/sql/scalar_functions.html#make-array)) that results in a single column output of `UnionArray`

```sql
--- create a single UNION with two members (String and F64) with the string variant
select make_union('a'::varchar, null::double)

--- create a single UNION with two members (String and F64) with the f64
select make_union(null::varchar, 1.234)

--- Error: can't figure out which variant to create
select make_union('a', 123.4)

```

An alternate, option would be to register tables specially for similarly to https://github.com/apache/datafusion/blob/16e3831734358d2c628c7ff281cddd680dc4aa10/datafusion/sqllogictest/src/test_context.rs#L75-L78)

We could create a UnionArray using the examples from https://docs.rs/arrow/latest/arrow/array/struct.UnionArray.html

### Additional context

See https://github.com/apache/datafusion/pull/10139#pullrequestreview-2013390566

Contributor guide

Open the contributing guide

Research direction

Start with the proposed union_type.slt and compare it with the existing union.slt. Read the sqllogictest test_context.rs registration area and the Arrow UnionArray examples to evaluate the suggested approaches. Done means DataType::Union has executable sqllogictest coverage, including the supported variants and invalid or ambiguous cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
backend, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.