apache / apache/beam

Can't use java logical types defined in Java JDBC IO in XLang JDBC pipeline for Go and Python

Open
#21,433 1 comment 0 reactions 0 assignees View on GitHub
bug io java jdbc P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Currently, we can't use the logical types defined in [java JDBC IO](https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/LogicalTypes.java) in XLang pipelines for JDBC IO in Go SDK as it throws the following error:
```

  unmarshalling coder jFYvjSCanTRowCoder_length_prefixed_windowed
        unmarshalling coder jFYvjSCanTRowCoder_length_prefixed
cannot
convert schema field role_name to field
        caused by:
unknown logical type: beam:logical_type:javasdk:v1

```

This is the structure of table and its corresponding struct:
```

// roles=> CREATE TABLE roles(
//         role_id serial PRIMARY KEY,
//         role_name VARCHAR
(255) UNIQUE
//  );

type JdbcTestRow struct {
        Role_id   int32  `beam:"role_id"`
     
  Role_name string `beam:"role_name"`
}
```

Similar error seems to be appearing on Python SDK pipeline based on this stack overflow question: ([https://stackoverflow.com/questions/68758361/error-beamlogical-typejavasdkv1-while-using-apache-beam-io-jdbc-readfromjdbc](https://stackoverflow.com/questions/68758361/error-beamlogical-typejavasdkv1-while-using-apache-beam-io-jdbc-readfromjdbc))

Imported from Jira [BEAM-13717](https://issues.apache.org/jira/browse/BEAM-13717). Original Jira may contain additional context.
Reported by: riteshghorse.

Contributor guide

Open the contributing guide

Research direction

Start with sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/LogicalTypes.java and reproduce the JDBC read using the shown roles table and Go or Python row structure. Trace the coder unmarshalling failure for beam:logical_type:javasdk:v1, and consult the linked BEAM-13717 Jira issue for additional context. Done means the logical type can be used by the XLang JDBC pipeline without the unknown logical type error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, java, python
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.