confluentinc / confluentinc/ksql

KSQL should support STRUCT fields for TIMESTAMP

Open
#2,188 6 comments 20 reactions 0 assignees View on GitHub
data-accessibility enhancement P1 streaming-engine user-experience
Dominant language
Java
Stars
315
Forks
1k
Avg merge
1d 7h
Merged PRs (30d)
36

Description

KSQL only supports top-level fields in the `WITH (TIMESTAMP` clause when defining a stream against an existing topic. It should also support nested fields.

For example, given source message

{"type":"key1","data":{"ts":"2018-11-20 19:20:21.1","a":1,"b":"hello"}}

it should be possible to execute:

```
CREATE STREAM T2 (TYPE VARCHAR, \
DATA STRUCT< \
TS VARCHAR, \
A INT, \
B VARCHAR, \
C INT, \
D VARCHAR>) \
WITH (KAFKA_TOPIC='t_raw',\
VALUE_FORMAT='JSON', TIMESTAMP='DATA->TS', TIMESTAMP_FORMAT='yyyy-MM-dd HH:mm:ss.S');
```

but it fails:

No column with the provided timestamp column name in the WITH clause, DATA->TS, exists in the defined schema.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the CREATE STREAM example with the nested DATA->TS timestamp expression and inspect how the TIMESTAMP clause resolves fields in the defined schema. Trace the validation from the CREATE STREAM entry point; done means the nested timestamp field is accepted and the example executes without the reported missing-column error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, sql
Domain
databases, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.