confluentinc / confluentinc/ksql
JOIN followed by PARTITION BY adds pseudo columns to projection
- Dominant language
- Java
- Stars
- 315
- Forks
- 1k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 36
Description
**Describe the bug**
As proposed in https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-14-rowtime-as-pseudocolumn.md, pseudo columns such as ROWTIME should not be included in `SELECT *` projections. This is true for both stream-stream joins ([1](https://github.com/confluentinc/ksql/blob/8f588d35b02799a0e29d929cdddb56d93fb49101/ksqldb-functional-tests/src/test/resources/query-validation-tests/joins.json#L4), [2](https://github.com/confluentinc/ksql/blob/8f588d35b02799a0e29d929cdddb56d93fb49101/ksqldb-functional-tests/src/test/resources/query-validation-tests/joins.json#L50)) and queries that involve PARTITION BY ([3](https://github.com/confluentinc/ksql/blob/8f588d35b02799a0e29d929cdddb56d93fb49101/ksqldb-functional-tests/src/test/resources/query-validation-tests/partition-by.json#L4), [4](https://github.com/confluentinc/ksql/blob/8f588d35b02799a0e29d929cdddb56d93fb49101/ksqldb-functional-tests/src/test/resources/query-validation-tests/partition-by.json#L52)), but not true for queries that involve both: https://github.com/confluentinc/ksql/blob/8f588d35b02799a0e29d929cdddb56d93fb49101/ksqldb-functional-tests/src/test/resources/query-validation-tests/partition-by.json#L240-L257
**To Reproduce**
See QTT above. Bug appears to have been present ever since the new semantics for pseudo columns were introduced, and is still present on master.
**Expected behavior**
The output schema of a query such as
```
CREATE STREAM OUTPUT AS SELECT * FROM L JOIN R WITHIN 10 SECONDS ON L.A = R.X PARTITION BY L.B;
```
(from the example QTT above) should not contain `L_ROWTIME BIGINT` and `R_ROWTIME BIGINT`.
**Actual behaviour**
Output schema contains the pseudo columns mentioned above.
**Additional context**
N/A
Contributor guide
Research direction
Start with the JOIN followed by PARTITION BY case in ksqldb-functional-tests/src/test/resources/query-validation-tests/partition-by.json, around lines 240-257, and compare it with the existing JOIN and PARTITION BY cases linked in the issue. Run that QTT and trace the output schema for SELECT *; done means L_ROWTIME and R_ROWTIME are absent while the other expected columns remain.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases, stream-processing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100