New `gcp_big_query` sink

GCP Big Query is a powerful service for analyzing large amounts of structured data. If used correctly, it can be a cost-effective storage for log data. I would like to see Vector support this service as a sink, but it'll require careful planning due to the many different ways Big Query can be used.

Context

Big Query is flexible and we should consider the following features for our Big Query sink:

1. Storing JSON data in a single column. Users can use Big Query's JSON functions.
   1. This type of querying is slower for obvious reasons (fetching more data, parsing, etc).
   2. This type of querying is more expensive because each query must fetch and scan the entire JSON payload as opposed to individual columns.
2. Mapping structured data to individual Big Query table columns.
3. Automatic schema maintenance.
4. Streaming inserts vs batching inserts.
   1. Streaming inserts have a cost ($0.010 per 200 MB).
   2. Note the variety of methods to batch insert. This can be done through the API directly or through other GCP services (cloud storage, stackdriver, etc).

This, of course, is not inclusive of all factors we should consider for Big Query, but it helps to demonstrate the variety of options.

Starting simple

v1 of this sink should solve the simplest implementation:

• Use the streaming inserts API and stream records 1 by 1 (no batching).
• Assume we are writing to a table with 2 columns: timestamp and json_event.
  • The timestamp column should map to our own internal timestamp column.
  • The json_event column should contain a JSON encoded representation of our event.
  • Both of these column names should be configurable (defaulting to the global log_schema.timestamp_key). It is worth thinking about a generic column mapping configuration scheme so that users could map other custom fields to Big Query columns.
• Include documentation on how to create a properly structured table. Ideally this table would be partitioned by the timestamp day.

Long Term

We might consider the following features for long-term development:

1. Support for using the batching API since it does not incur a cost.
2. Dynamic schema maintenance. Although, I think this might be better solved with a transform or something separate.

Review the streaming inserts API and the v1 checklist, especially the configurable timestamp and json_event column mappings. Done means a working one-record-at-a-time sink, support for the two-column table shape, and documentation for creating a timestamp-partitioned table.