apache / apache/arrow-rs

Avro: the default value for a union-typed field should be accepted regardless of the ordinal of the matching schema variant (spec 1.12)

Open
#9,304 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Describe the bug**

[The logic](https://github.com/apache/arrow-rs/blob/2c0eba46ec447d815ddc7f8185edbedd2ae3596b/arrow-avro/src/codec.rs#L312) of handling a `null` default value in a field schema where the type is a union only allows the null variant to be listed first. The comment above claims this to be per-spec, but the specification says only this, as of version 1.12.0:

> Default values for union fields correspond to the first schema that matches in the union.

There is no restriction that the default value must match the first schema variant.

**To Reproduce**

An application uses an Avro schema (e.g. as the reader schema) specifying a field like:

```json
{
"name": "optional_int",
"type": ["int", "null"],
"default": null
}
```

Building the reader fails with an error:
> Schema error: JSON null default is only valid for `null` type or for a union whose first branch is `null`

**Expected behavior**

The default should be accepted because it matches one of the union schemas, as per [the Avro spec](https://avro.apache.org/docs/1.12.0/specification/#schema-record) version 1.12.0.

**Additional context**

An example of this order preference is [found](https://github.com/apache/spark/blob/v3.5.8/connector/avro/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala#L277) in Spark 3.5.

Contributor guide

Open the contributing guide

Research direction

Start in arrow-avro/src/codec.rs at the linked logic around line 312 and reproduce the failure with a reader schema whose union is ["int", "null"] and default is null. Compare the validation with the Avro 1.12 rule that the default matches a union branch, then add coverage showing the schema is accepted and run the relevant arrow-avro tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.