apache / apache/wayang

Missing schema/type validation between SQL and Java operators leads to runtime ClassCastException

Open
#747 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
274
Forks
140
Avg merge
5d 16h
Merged PRs (30d)
4

Description

While working on a pipeline like PostgresTableSource → Join → SqlToStream → Java map → collect, I faced a runtime ClassCastException. The issue is happening because after the SQL join, the data is actually in a tuple-like structure (like Tuple2), but during the SqlToStream conversion, this structure is not preserved properly and everything is treated as a Record.

Because of this, when I try to use the data in the Java map step (for example accessing fields or casting types), it assumes a wrong structure and crashes at runtime (e.g., String cannot be cast to Integer). There is no validation or check between SQL and Java stages, so the mismatch is not caught early and only fails during execution.

Ideally, the correct schema/type should be preserved (or validated) when moving from SQL to Java, so that such issues don’t happen at runtime.

for example -

CREATE TABLE table_a (
id INT,
name TEXT
);

CREATE TABLE table_b (
id TEXT, -- different type
value TEXT
);

Image

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the PostgresTableSource → Join → SqlToStream → Java map pipeline described in the issue, comparing the join output with the structure received by the Java stage. Trace where the tuple-like result is converted or its schema is lost. Done means incompatible SQL and Java types are preserved or rejected before execution rather than producing a runtime ClassCastException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.