Avro: the default value for a union-typed field should be accepted regardless of the ordinal of the matching schema variant (spec 1.12)
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Describe the bug**
[The logic](https://github.com/apache/arrow-rs/blob/2c0eba46ec447d815ddc7f8185edbedd2ae3596b/arrow-avro/src/codec.rs#L312) of handling a `null` default value in a field schema where the type is a union only allows the null variant to be listed first. The comment above claims this to be per-spec, but the specification says only this, as of version 1.12.0:
> Default values for union fields correspond to the first schema that matches in the union.
There is no restriction that the default value must match the first schema variant.
**To Reproduce**
An application uses an Avro schema (e.g. as the reader schema) specifying a field like:
```json
{
"name": "optional_int",
"type": ["int", "null"],
"default": null
}
```
Building the reader fails with an error:
> Schema error: JSON null default is only valid for `null` type or for a union whose first branch is `null`
**Expected behavior**
The default should be accepted because it matches one of the union schemas, as per [the Avro spec](https://avro.apache.org/docs/1.12.0/specification/#schema-record) version 1.12.0.
**Additional context**
An example of this order preference is [found](https://github.com/apache/spark/blob/v3.5.8/connector/avro/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala#L277) in Spark 3.5.
Contributor guide
Research direction
Start in arrow-avro/src/codec.rs at the linked logic around line 312 and reproduce the failure with a reader schema whose union is ["int", "null"] and default is null. Compare the validation with the Avro 1.12 rule that the default matches a union branch, then add coverage showing the schema is accepted and run the relevant arrow-avro tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100