GoogleCloudPlatform / GoogleCloudPlatform/market-data-transcoder

Add capability to shard message output based on values found in messages

Open
#67 0 comments 0 reactions 0 assignees View on GitHub
enhancement feedback wanted
Dominant language
Python
Stars
47
Forks
11
Avg merge
19d 10h
Merged PRs (30d)
2

Description

Currently, the transcoder emits messages to output sources based on the message type specified in the schema. This would add a capability to further filter messages based on input dimensions observed, sharding them to separate outputs. For example, if you specify a list of `--shard_keys` fully qualified with the parent message type, it would manufacture a file/table/topic for each distinct value found in that dimension. So

```--shard_keys SnapshotFullRefreshTcpLongQty.SecurityID```

would produce and populate a separate persistent resource for each unique value of that dimension found in the input data.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.