apache / apache/beam

[Feature Request]: Design and Implement CoderLogicalType

Open
#23,374 1 comment 0 reactions 0 assignees View on GitHub
cross-language new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

It was mentioned in #7865 that a CoderLogicalType could enable any existing Coder to be used as a Schema field. In particular, the micros_millis and decimal logical types are made to be compatible with existing datetime and decimal java fieldtype, and they do not fit in the beam portable logical type representations. Currently they are treated as special cases with a representation that is not quite true: https://github.com/apache/beam/blob/3a4d57eb8976c5f503b32d478a80b1800490f66f/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaTranslation.java#L201

For example,
here DATETIME has a representation of FieldType.INT64, but FieldType.INT64 is decoded with VarIntCoder, different from the actually used BigEndianLongCoder.
here DECIMAL has a representation of FieldType.BYTES, but the actually used BigDecimalCoder encoded a VarInt and a Bytes.

Introducing CoderLogicalType and specify the coder to the logical type's payload could clarify this issue.

### Issue Priority

Priority: 2

### Issue Component

Component: cross-language

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.