confluentinc / confluentinc/ksql
KSQL should support STRUCT fields for TIMESTAMP
- Dominant language
- Java
- Stars
- 315
- Forks
- 1k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 36
Description
KSQL only supports top-level fields in the `WITH (TIMESTAMP` clause when defining a stream against an existing topic. It should also support nested fields.
For example, given source message
{"type":"key1","data":{"ts":"2018-11-20 19:20:21.1","a":1,"b":"hello"}}
it should be possible to execute:
```
CREATE STREAM T2 (TYPE VARCHAR, \
DATA STRUCT< \
TS VARCHAR, \
A INT, \
B VARCHAR, \
C INT, \
D VARCHAR>) \
WITH (KAFKA_TOPIC='t_raw',\
VALUE_FORMAT='JSON', TIMESTAMP='DATA->TS', TIMESTAMP_FORMAT='yyyy-MM-dd HH:mm:ss.S');
```
but it fails:
No column with the provided timestamp column name in the WITH clause, DATA->TS, exists in the defined schema.
Contributor guide
Research direction
Start by reproducing the CREATE STREAM example with the nested DATA->TS timestamp expression and inspect how the TIMESTAMP clause resolves fields in the defined schema. Trace the validation from the CREATE STREAM entry point; done means the nested timestamp field is accepted and the example executes without the reported missing-column error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka, sql
- Domain
- databases, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100