[Bug]: Poor SQL -> PTransform translation in SqlTransform
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
When looking at [TPC-DS query 83](https://github.com/apache/beam/blob/883a362c930aca4298551697d7aaacbe7b6602f1/sdks/java/testing/tpcds/src/main/resources/queries/query83.sql) I noticed that the generated `PTransform` is extremely complex and not optimized as it should be.
One issue I observed is that we're exploding / denormalizing the `RelNode` graph during the conversion to `BeamRelNode`s.
For instance, in the case of query 83 this is happening for this subquery.
```
select d_week_seq
from date_dim
where d_date in ('1998-01-02','1998-10-15','1998-11-10')
```
Initially the repeated reference was normalized into a single `RelNode`. Though, when applying the conversion rules we might end up creating [multiple copies](https://github.com/apache/calcite/blob/e37ecb63e0c59aec7a4ba79b4d3473dd1cd2cfed/core/src/main/java/org/apache/calcite/plan/volcano/RelSubset.java#L734-L739), leading to independent PTransforms of the same "SQL" for each occurrence.
Below a visualization of the denormalized transform hierarchy for TPC-DS query 83, followed by a similar one using [a hack](https://github.com/mosche/beam/commit/874fa34e7ed8567678443775adf520d326a30243) to keep it in its normalized shape.
### "Denormalized" transform hierarchy

### "Normalized" transform hierarchy

### Issue Priority
Priority: 2
### Issue Component
Component: dsl-sql
Contributor guide
Assessment
This issue has not been assessed yet.