aws-samples / aws-samples/pyflink-getting-started

data generation code - stock.py - could be improved by changing partition key

Open
#26 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
53
Forks
26
PR merge metrics
No merged PRs in 30d

Description

I was stuck on testing FileSink example code to read data from kinesis and then write to S3. It ran without any error, but there was no file on S3. Eventually I figured out that was because stock.py was using fixed 'partitionkey' as partition key, and if there were more than one shard in Kenisis, the data was only written to one shard. However the window function was waiting for data from other shards, thus the window computing was never done.

I was using table.exec.source.idle-timeout configuration to conquer this problem initially. Then I think the code stock.py could be improved by a small change on line 24:

PartitionKey=data['event_time'][-2:])

That will use the last two digits of event time as partition key.

Contributor guide

Open the contributing guide

Research direction

Open stock.py and inspect line 24, then read the FileSink example flow from Kinesis to S3. Confirm how the generated event time is used as the partition key and verify the example can process records across multiple shards and produce output in S3.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
stream-processing
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.