linkedin / linkedin/goavro

CanonicalSchema() is non-deterministic when a field name matches a record type name in the same namespace

Open
#307 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.1k
Forks
232
PR merge metrics
No merged PRs in 30d

Description

## Summary

`CanonicalSchema()` produces two different output strings from the same schema input, non-deterministically (~14% vs ~86% split), when a **field name** and a **record type name** share the same local name within the same inherited namespace.

## Reproduction

```go
package main

import (
"fmt"
"github.com/linkedin/goavro/v2"
)

const schema = `{
"type": "record",
"name": "Parent",
"namespace": "com.example",
"fields": [{
"name": "items",
"type": ["null", {
"type": "array",
"items": {
"type": "record",
"name": "items",
"fields": [{"name": "id", "type": "int"}]
}
}],
"default": null
}]
}`

func main() {
seen := map[string]int{}
for i := 0; i < 1000; i++ {
codec, _ := goavro.NewCodec(schema)
seen[codec.CanonicalSchema()]++
}
fmt.Printf("distinct canonical forms: %d\n", len(seen))
for k, v := range seen {
fmt.Printf(" count=%-4d %s\n", v, k)
}
}
```

**Output (example run):**

```
distinct canonical forms: 2
count=862 {"name":"com.example.Parent","type":"record","fields":[{"name":"items","type":["null",{"type":"array","items":{"name":"com.example.items","type":"record","fields":[{"name":"id","type":"int"}]}}]}]}
count=138 {"name":"com.example.Parent","type":"record","fields":[{"name":"com.example.items","type":["null",{"type":"array","items":{"name":"com.example.items","type":"record","fields":[{"name":"id","type":"int"}]}}]}]}
```

The difference is the `"name"` of the `items` field on the outer `Parent` record:

- **Correct (~86%):** `"name":"items"` — the field's declared local name
- **Wrong (~14%):** `"name":"com.example.items"` — incorrectly namespace-qualified

## Expected behaviour

`CanonicalSchema()` should always produce the same output for the same input schema. Per the [Avro Parsing Canonical Form spec](https://avro.apache.org/docs/current/spec.html#Parsing+Canonical+Form+for+Schemas), field names are never namespace-qualified — the second form is incorrect.

## Root cause

goavro builds an internal map of named types during schema parsing. Go map iteration is non-deterministic. When computing the canonical form for a field, the code appears to consult this map to resolve whether the field name is a type reference, and non-deterministic iteration occasionally causes the field's local name to be replaced with the fully-qualified type name of the same-named record.

The field `"name": "items"` collides with the record type `com.example.items`. When goavro's named-type lookup happens to find that match during canonical form computation, it emits `"com.example.items"` as the field name instead of `"items"`.

## Impact

Any system that uses `CanonicalSchema()` output as a fingerprint (e.g. to compare schemas between a registry and a stored file) will see spurious mismatches ~14% of the time when this collision exists. We encountered this when validating exported Avro OCF files against a Pub/Sub schema registry — ~22 out of 135 daily files were flagged as mismatched despite containing identical schemas.

## Workaround

Rename the record type (or field) so they no longer share a local name. This makes `CanonicalSchema()` deterministic again.

## Environment

- goavro version: v2.15.0 (latest)
- Go version: 1.25.4
- OS: darwin/arm64 (also reproducible on linux/amd64 in CI)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at CanonicalSchema() and the schema-parsing code that builds the named-type map, using the supplied reproduction to trigger the field/type-name collision. Add a regression test showing repeated canonicalization is deterministic and that the outer field remains named "items"; done means every output matches the correct canonical form.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.