Azure / Azure/Azure-DataFactory

Avro dataset: nullable fields (unions) are in reverse order

Open
#189 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

The automatically generated Avro schema contained in the OCF header of files from ADF are unions of ` | null`, rather than `null | `, which does not follow the recommendation in the Avro spec for nullable types:

> (Note that when a default value is specified for a record field whose type is a union, the type of the default value must match the first element of the union. Thus, for unions containing "null", the "null" is usually listed first, since the default value of such unions is typically null.)

https://avro.apache.org/docs/1.8.1/spec.html#Unions

The outcome is that some libraries fail to identify these fields as nullable types, for example in Go where a nullable union of T can be mapped to a (nullable) pointer *T.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.