kestra-io / kestra-io/plugin-serdes

fix(avro): a real string listed in nullValues is fabricated into a null for ["null", T] unions

Open
#397 0 comments 0 reactions 0 assignees View on GitHub
area/plugin good first issue
Dominant language
Java
Stars
1
Forks
20
Avg merge
1d 18h
Merged PRs (30d)
36

Description

### Problem

`AvroConverter.primitiveNull` turns any string listed in `nullValues` into a real Avro `null`, regardless of `inferAllFields`. For a `["null", T]` union — the branch order `InferAvroSchema` emits for *every* nullable field — the `NULL` branch is tried first, so a genuine user string that happens to be in `nullValues` is fabricated into a null.

Measured with `inferAllFields: false` (the default):

```
["null","string"] + literal "null" -> real Avro null <- data destroyed
["null","string"] + literal "n/a" -> real Avro null <- data destroyed
["null","string"] + literal "NA" -> real Avro null <- data destroyed
["null","string"] + literal "" -> real Avro null <- data destroyed
["string","null"] + literal "null" -> String("null") <- preserved
plain "string" + literal "null" -> String("null") <- preserved
```

Two things are wrong here:

1. **Real values are silently replaced by nulls.** This is the mirror image of #368 (where a null was fabricated into the string `"null"`), and carries the same "hides that the data was ever changed" risk: the task returns SUCCESS with no warning, and `onBadLines` is irrelevant because no conversion failure is raised.
2. **The same Avro type yields different data depending on union branch order.** `["null","string"]` and `["string","null"]` are the same logical type, and should not disagree about what the input meant. Since `InferAvroSchema` always emits `["null", T]`, every inferred-schema conversion sits in the affected branch order.

The `nullValues` documentation on `AbstractAvroConverter` (line 47) says it is "Ignored for schema inference when `inferAllFields` is `true`", which reads as if the mapping is tied to inference. It is not — the mapping in `primitiveNull` is unconditional.

### Reproduction

`AvroConverter.java:514`:

```java
protected Integer primitiveNull(Object data) {
if (data instanceof String && this.contains(this.getNullValues(), (String) data)) {
return null; // <- applies with inferAllFields: false too
} else if (data == null) {
return null;
} else {
throw new IllegalArgumentException("Unknown type for null values, found " + data.getClass().getName());
}
}
```

`ComplexUnionTest` already demonstrates the behaviour on `main` (line 22). It predates #368 and PR #392 — the new `PrimitiveStringBytesTest` guard-rail case only covers plain `STRING`/`BYTES`, where the mapping never applies, so it does not catch this.

### Expected

A string-typed union branch should win over the `NULL` branch for a non-empty string value, independent of the order the branches are declared in. At minimum, `primitiveNull` should not apply the `nullValues` mapping when a string branch is available in the same union.

Whatever the resolution, the two orderings must agree, and `nullValues`' documentation should state exactly when the mapping applies.

### Context

Found during peer review of #392 (fix for #368) by @Malaydewangan09, and verified independently against the code.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at AvroConverter.java:514 and inspect how primitiveNull is reached for nullable unions. Reproduce the behavior in ComplexUnionTest, then add coverage showing ["null","string"] and ["string","null"] preserve the same non-empty string values. Review AbstractAvroConverter line 47 and update the nullValues documentation to match the resolved behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.