Can't use java logical types defined in Java JDBC IO in XLang JDBC pipeline for Go and Python
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Currently, we can't use the logical types defined in [java JDBC IO](https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/LogicalTypes.java) in XLang pipelines for JDBC IO in Go SDK as it throws the following error:
```
unmarshalling coder jFYvjSCanTRowCoder_length_prefixed_windowed
unmarshalling coder jFYvjSCanTRowCoder_length_prefixed
cannot
convert schema field role_name to field
caused by:
unknown logical type: beam:logical_type:javasdk:v1
```
This is the structure of table and its corresponding struct:
```
// roles=> CREATE TABLE roles(
// role_id serial PRIMARY KEY,
// role_name VARCHAR
(255) UNIQUE
// );
type JdbcTestRow struct {
Role_id int32 `beam:"role_id"`
Role_name string `beam:"role_name"`
}
```
Similar error seems to be appearing on Python SDK pipeline based on this stack overflow question: ([https://stackoverflow.com/questions/68758361/error-beamlogical-typejavasdkv1-while-using-apache-beam-io-jdbc-readfromjdbc](https://stackoverflow.com/questions/68758361/error-beamlogical-typejavasdkv1-while-using-apache-beam-io-jdbc-readfromjdbc))
Imported from Jira [BEAM-13717](https://issues.apache.org/jira/browse/BEAM-13717). Original Jira may contain additional context.
Reported by: riteshghorse.
Contributor guide
Research direction
Start with sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/LogicalTypes.java and reproduce the JDBC read using the shown roles table and Go or Python row structure. Trace the coder unmarshalling failure for beam:logical_type:javasdk:v1, and consult the linked BEAM-13717 Jira issue for additional context. Done means the logical type can be used by the XLang JDBC pipeline without the unknown logical type error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, java, python
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100