apache / apache/beam

optional parameter is not supported in BeamSQL

Open
#18,672 0 comments 0 reactions 0 assignees View on GitHub
bug dsl P3 sql
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

I'm wring a UDF with optional parameters, the method code is:
```

public static String eval(String kvString, String keyName
, @Parameter(name = "pd", optional
= true) String pairSeperator
, @Parameter(name = "kvd", optional = true) String kvSeperator){

//...
}

```

And I see this error when assembling a Beam pipeline:
```

Execution Plan:
BeamProjectRel(ITEM_ID=[CAST(KV_EXT($0, 'itemId', DEFAULT(), DEFAULT())):BIGINT],
TRANSACTION_ID=[CAST(KV_EXT($0, 'transactionId', DEFAULT(), DEFAULT())):BIGINT], TRANSACTION_TIME=[KV_EXT($0,
'createdDT', DEFAULT(), DEFAULT())])
BeamIOSourceRel(table=[[....]])

Exception in thread "main"
java.lang.RuntimeException: java.lang.UnsupportedOperationException: Operator: DEFAULT is not supported
yet!
at .........
Caused by: java.lang.UnsupportedOperationException: Operator: DEFAULT is not supported
yet!
at org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:425)
at
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:202)
at
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:202)
at
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.(BeamSqlFnExecutor.java:126)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamProjectRel.buildBeamPipeline(BeamProjectRel.java:70)
at
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.compileBeamPipeline(BeamQueryPlanner.java:116)
at
org.apache.beam.sdk.extensions.sql.BeamSqlCli.compilePipeline(BeamSqlCli.java:112)

```

Imported from Jira [BEAM-3325](https://issues.apache.org/jira/browse/BEAM-3325). Original Jira may contain additional context.
Reported by: mingmxu.

Contributor guide

Open the contributing guide

Research direction

Start at BeamSqlFnExecutor.java:425, where DEFAULT is rejected, and trace how the expression is reached from BeamProjectRel and BeamQueryPlanner. Determine how optional UDF parameters should be assembled, then verify that the shown BeamSQL pipeline compiles without the unsupported DEFAULT error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.