kestra-io / kestra-io/plugin-serdes
fix(avro): a real string listed in nullValues is fabricated into a null for ["null", T] unions
- Dominant language
- Java
- Stars
- 1
- Forks
- 20
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 36
Description
### Problem
`AvroConverter.primitiveNull` turns any string listed in `nullValues` into a real Avro `null`, regardless of `inferAllFields`. For a `["null", T]` union — the branch order `InferAvroSchema` emits for *every* nullable field — the `NULL` branch is tried first, so a genuine user string that happens to be in `nullValues` is fabricated into a null.
Measured with `inferAllFields: false` (the default):
```
["null","string"] + literal "null" -> real Avro null <- data destroyed
["null","string"] + literal "n/a" -> real Avro null <- data destroyed
["null","string"] + literal "NA" -> real Avro null <- data destroyed
["null","string"] + literal "" -> real Avro null <- data destroyed
["string","null"] + literal "null" -> String("null") <- preserved
plain "string" + literal "null" -> String("null") <- preserved
```
Two things are wrong here:
1. **Real values are silently replaced by nulls.** This is the mirror image of #368 (where a null was fabricated into the string `"null"`), and carries the same "hides that the data was ever changed" risk: the task returns SUCCESS with no warning, and `onBadLines` is irrelevant because no conversion failure is raised.
2. **The same Avro type yields different data depending on union branch order.** `["null","string"]` and `["string","null"]` are the same logical type, and should not disagree about what the input meant. Since `InferAvroSchema` always emits `["null", T]`, every inferred-schema conversion sits in the affected branch order.
The `nullValues` documentation on `AbstractAvroConverter` (line 47) says it is "Ignored for schema inference when `inferAllFields` is `true`", which reads as if the mapping is tied to inference. It is not — the mapping in `primitiveNull` is unconditional.
### Reproduction
`AvroConverter.java:514`:
```java
protected Integer primitiveNull(Object data) {
if (data instanceof String && this.contains(this.getNullValues(), (String) data)) {
return null; // <- applies with inferAllFields: false too
} else if (data == null) {
return null;
} else {
throw new IllegalArgumentException("Unknown type for null values, found " + data.getClass().getName());
}
}
```
`ComplexUnionTest` already demonstrates the behaviour on `main` (line 22). It predates #368 and PR #392 — the new `PrimitiveStringBytesTest` guard-rail case only covers plain `STRING`/`BYTES`, where the mapping never applies, so it does not catch this.
### Expected
A string-typed union branch should win over the `NULL` branch for a non-empty string value, independent of the order the branches are declared in. At minimum, `primitiveNull` should not apply the `nullValues` mapping when a string branch is available in the same union.
Whatever the resolution, the two orderings must agree, and `nullValues`' documentation should state exactly when the mapping applies.
### Context
Found during peer review of #392 (fix for #368) by @Malaydewangan09, and verified independently against the code.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at AvroConverter.java:514 and inspect how primitiveNull is reached for nullable unions. Reproduce the behavior in ComplexUnionTest, then add coverage showing ["null","string"] and ["string","null"] preserve the same non-empty string values. Review AbstractAvroConverter line 47 and update the nullValues documentation to match the resolved behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100