apache / apache/beam

[Bug]: condition missed after a unnest

Open
#27,733 5 comments 0 reactions 0 assignees View on GitHub
bug flink java P2 sql stale
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

```java
Schema schema = Schema.builder()
.addInt64Field("id")
.addStringField("json")
.build();

Schema outputSchema = Schema.builder()
.addInt64Field("id")
.addStringField("json")
.addStringField("json_element")
.build();

Row row = Row.withSchema(schema)
.addValue(System.currentTimeMillis())
.addValue("[{\"id\":1},{\"id\":1123456789},{\"id\":-1}]")
.build();

Pipeline pipeline = Pipeline.create(PipelineOptionsFactory.fromArgs("--runner=FlinkRunner"
, "--fasterCopy=true"
, "--disableMetrics=true"
, "--externalizedCheckpointsEnabled=false").create());

pipeline.apply(Create.of(row))
.setRowSchema(schema)
.apply("filter", SqlTransform
.query("select * from" +
" (select a.*,j.i from PCOLLECTION a" +
" cross join unnest(to_json_str_array(json)) as j(i)" +
") where json_value(i,'$.id') <> -1 "))
.apply("output", MapElements.via(new SimpleFunction() {
@Override
public Row apply(Row input) {
logger.info("current row {}", input);
return input;
}
}))
.setRowSchema(outputSchema);
pipeline.run().waitUntilFinish();
```

Hi, folks, I'm trying to filter the final result using a simple condition, however it doesn't works. All data are outputed in final result, can you help to look into it?

```java
2023-07-28 15:46:32.772 org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner INFO - SQLPlan>
LogicalProject(id=[$0], json=[$1], i=[$2])
LogicalProject(id=[$0], json=[$1], i=[$2])
LogicalProject(id=[$0], json=[$1], i=[$3])
LogicalCorrelate(correlation=[$cor0], joinType=[inner], requiredColumns=[{2}])
LogicalProject(id=[$0], json=[$1], $f2=[to_json_str_array($1)])
BeamIOSourceRel(table=[[beam, PCOLLECTION]])
LogicalFilter(condition=[<>(CAST(JSON_VALUE($0, '$.id')):INTEGER, -1)])
LogicalProject(i=[$0])
Uncollect
LogicalProject(EXPR$0=[$cor0.$f2])
LogicalValues(tuples=[[{ 0 }]])

2023-07-28 15:46:32.964 org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner INFO - BEAMPlan>
BeamCalcRel(expr#0..3=[{inputs}], proj#0..1=[{exprs}], i=[$t3])
BeamUnnestRel(unnestIndices=[[2]])
BeamCalcRel(expr#0..1=[{inputs}], expr#2=[to_json_str_array($t1)], proj#0..2=[{exprs}])
BeamIOSourceRel(table=[[beam, PCOLLECTION]])
```

It seems that `LogicalFilter(condition=[<>(CAST(JSON_VALUE($0, '$.id')):INTEGER, -1)])` is missing

### Issue Priority

Priority: 0 (outage / urgent vulnerability)

### Issue Components

- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [X] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.