GoogleCloudPlatform / GoogleCloudPlatform/market-data-transcoder

Schemas with repeating values at different levels in the message graph are incompatible with Avro

Open
#73 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
47
Forks
11
Avg merge
19d 10h
Merged PRs (30d)
2

Description

When running with the following set of options:

```
wget -q -O - ftp://ftp.cmegroup.com/SBEFix/Production/secdef.dat.gz | gunzip - | txcode \
--schema_file ~/src/datacast/transcoder/test/FIX50SP2.CME.xml \
--factory fix \
--source_file_format_type line_delimited \
--message_type_inclusions=SecurityDefinition \
--fix_header_tags 8,9,35,1128,49,56,34,52,10 \
--destination_project_id $(gcloud config get-value project) \
--output_type pubsub \
--output_encoding json \
--lazy_create_resources
```

The following error is thrown:

```
google.api_core.exceptions.InvalidArgument: 400 AVRO schema definition is not valid: sbeMessage.NoLotTypeRules exists twice in schema. [detail: "[ORIGINAL ERROR] generic::invalid_argument: AVRO schema definition is not valid: sbeMessage.NoLotTypeRules exists twice in schema. [google.rpc.error_details_ext] { message: \"AVRO schema definition is not valid: sbeMessage.NoLotTypeRules exists twice in schema.\" }"
```

This is likely because of the complex schema defined in `FIX50SP2.CME.xml`, where the same field name may be present at different levels of the entity hierarchy. A graph such as `Object.Property1` and `Object.Property2.Property1` appears to be incompatible with Avro, but is commonly encountered within legitimate FIX schema definitions.

It's notable that BigQuery output types do not exhibit this behavior, but the `fastavro` output type to local POSIX files *does* as well:

```
fastavro._schema_common.SchemaParseException: redefined named type: sbeMessage.NoLotTypeRules
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.