apache / apache/beam

[Bug]: Poor SQL -> PTransform translation in SqlTransform

Open
#24,314 7 comments 0 reactions 0 assignees View on GitHub
bug dsl P2 performance sql tpcds
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

When looking at [TPC-DS query 83](https://github.com/apache/beam/blob/883a362c930aca4298551697d7aaacbe7b6602f1/sdks/java/testing/tpcds/src/main/resources/queries/query83.sql) I noticed that the generated `PTransform` is extremely complex and not optimized as it should be.

One issue I observed is that we're exploding / denormalizing the `RelNode` graph during the conversion to `BeamRelNode`s.
For instance, in the case of query 83 this is happening for this subquery.
```
select d_week_seq
from date_dim
where d_date in ('1998-01-02','1998-10-15','1998-11-10')
```

Initially the repeated reference was normalized into a single `RelNode`. Though, when applying the conversion rules we might end up creating [multiple copies](https://github.com/apache/calcite/blob/e37ecb63e0c59aec7a4ba79b4d3473dd1cd2cfed/core/src/main/java/org/apache/calcite/plan/volcano/RelSubset.java#L734-L739), leading to independent PTransforms of the same "SQL" for each occurrence.

Below a visualization of the denormalized transform hierarchy for TPC-DS query 83, followed by a similar one using [a hack](https://github.com/mosche/beam/commit/874fa34e7ed8567678443775adf520d326a30243) to keep it in its normalized shape.

### "Denormalized" transform hierarchy
![query83result1669102893010 dot](https://user-images.githubusercontent.com/1401430/203382452-d201b5ee-9e02-4368-9e51-b871d8d9315f.svg)

### "Normalized" transform hierarchy
![query83result1669137864884 dot](https://user-images.githubusercontent.com/1401430/203399026-1e3fa6d6-e17d-420e-8670-596adcb5a7c1.svg)

### Issue Priority

Priority: 2

### Issue Component

Component: dsl-sql

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.