GoogleCloudPlatform / GoogleCloudPlatform/market-data-transcoder
Add capability to shard message output based on values found in messages
- Dominant language
- Python
- Stars
- 47
- Forks
- 11
- Avg merge
- 19d 10h
- Merged PRs (30d)
- 2
Description
Currently, the transcoder emits messages to output sources based on the message type specified in the schema. This would add a capability to further filter messages based on input dimensions observed, sharding them to separate outputs. For example, if you specify a list of `--shard_keys` fully qualified with the parent message type, it would manufacture a file/table/topic for each distinct value found in that dimension. So
```--shard_keys SnapshotFullRefreshTcpLongQty.SecurityID```
would produce and populate a separate persistent resource for each unique value of that dimension found in the input data.
Contributor guide
Assessment
This issue has not been assessed yet.