apache / apache/beam

Queries that attempt to write to pubsub publish time should fail at construction time

Open
#19,875 0 comments 0 reactions 0 assignees View on GitHub
dsl improvement P3 sql
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Currently it's possible to perform a query like:

```

CREATE TABLE pubsub (
event_timestamp TIMESTAMP,
id VARCHAR
) ...

INSERT INTO pubsub (event_timestamp,
id) VALUES (...)

```

But when this is executed, the event_timestamp will be dropped, because on read it will be instead be populated with pubsub's publish time.

A couple of ideas:
- We could indicate that this is a VIRTUAL GENERATED column, and is therefore read-only. Calcite seems to have some support for this concept, see [ColumnStrategy.java](https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/schema/ColumnStrategy.java).
- We could just throw an exception in the Pubsub JSON Table Provider if the query's output schema contains event_timestamp.

Imported from Jira [BEAM-8741](https://issues.apache.org/jira/browse/BEAM-8741). Original Jira may contain additional context.
Reported by: bhulette.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.