influxdata / influxdata/telegraf

Auto create table in BigQuery

Open
#13,033 2 comments 0 reactions 0 assignees View on GitHub
area/gcp feature request help wanted plugin/output
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

With some input plugins the metric names and formats are unknown on beforehand, the number of tags and fields can vary over time. Therefore it would be nice if the `outputs.bigquery` plugin could handle this in a way similar to the `outputs.sql` plugin.

### Expected behavior

Telegraf creates the required table in BigQuery with all the corresponding metric fields and tags just as the sql

### Actual behavior

Tables for metrics need to be created on beforehand in BigQuery.

### Additional info

[Here is a PoC](https://gist.github.com/Hipska/1357083d666c5f57e2e0271159fab943) which adds a new BigQuery field on the fly, proving it should be possible.

Contributor guide

Open the contributing guide

Research direction

Review the outputs.bigquery plugin alongside outputs.sql and the linked PoC to understand how metric fields and tags should map to table creation. Verify completion with unknown metric schemas by confirming that the required BigQuery tables are created automatically.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud
Domain
cloud, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.