GoogleCloudPlatform / GoogleCloudPlatform/market-data-transcoder

Add option to interleave messages to an output destination

Open
#135 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
47
Forks
11
Avg merge
19d 10h
Merged PRs (30d)
2

Description

In lieu of sharding output destinations by message type, support interleaved output of all message types to a single destination file, topic, etc.

This has some implications to schema generation and most obviously would not work for BigQuery output types. It would also require that Pub/Sub topic schemas or Avro outputs be handled differently in this mode, perhaps using the Union option in Avro.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.