confluentinc / confluentinc/ksql

JOIN followed by PARTITION BY adds pseudo columns to projection

Open
#7,488 1 comment 0 reactions 0 assignees View on GitHub
bug streaming-engine
Dominant language
Java
Stars
315
Forks
1k
Avg merge
1d 7h
Merged PRs (30d)
36

Description

**Describe the bug**

As proposed in https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-14-rowtime-as-pseudocolumn.md, pseudo columns such as ROWTIME should not be included in `SELECT *` projections. This is true for both stream-stream joins ([1](https://github.com/confluentinc/ksql/blob/8f588d35b02799a0e29d929cdddb56d93fb49101/ksqldb-functional-tests/src/test/resources/query-validation-tests/joins.json#L4), [2](https://github.com/confluentinc/ksql/blob/8f588d35b02799a0e29d929cdddb56d93fb49101/ksqldb-functional-tests/src/test/resources/query-validation-tests/joins.json#L50)) and queries that involve PARTITION BY ([3](https://github.com/confluentinc/ksql/blob/8f588d35b02799a0e29d929cdddb56d93fb49101/ksqldb-functional-tests/src/test/resources/query-validation-tests/partition-by.json#L4), [4](https://github.com/confluentinc/ksql/blob/8f588d35b02799a0e29d929cdddb56d93fb49101/ksqldb-functional-tests/src/test/resources/query-validation-tests/partition-by.json#L52)), but not true for queries that involve both: https://github.com/confluentinc/ksql/blob/8f588d35b02799a0e29d929cdddb56d93fb49101/ksqldb-functional-tests/src/test/resources/query-validation-tests/partition-by.json#L240-L257

**To Reproduce**

See QTT above. Bug appears to have been present ever since the new semantics for pseudo columns were introduced, and is still present on master.

**Expected behavior**

The output schema of a query such as
```
CREATE STREAM OUTPUT AS SELECT * FROM L JOIN R WITHIN 10 SECONDS ON L.A = R.X PARTITION BY L.B;
```
(from the example QTT above) should not contain `L_ROWTIME BIGINT` and `R_ROWTIME BIGINT`.

**Actual behaviour**

Output schema contains the pseudo columns mentioned above.

**Additional context**

N/A

Contributor guide

Open the contributing guide

Research direction

Start with the JOIN followed by PARTITION BY case in ksqldb-functional-tests/src/test/resources/query-validation-tests/partition-by.json, around lines 240-257, and compare it with the existing JOIN and PARTITION BY cases linked in the issue. Run that QTT and trace the output schema for SELECT *; done means L_ROWTIME and R_ROWTIME are absent while the other expected columns remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases, stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.