apache / apache/beam

WriteToBigQuery: Schema auto-detection is not supported.

Open
#20,432 0 comments 0 reactions 0 assignees View on GitHub
bug gcp io P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

This is regarding the WriteToBigQuery connector which is supposed to support SCHEMA_AUTODETECTION but throws an error indicating that Schema auto-detection is not supported for streaming inserts into BigQuery (Only for File Loads) even when batching from a file.

SCHEMA_AUTODETECTION should be supported as per instructions (or consider adding more context to the instructions):  [https://beam.apache.org/releases/pydoc/2.15.0/_modules/apache_beam/io/gcp/bigquery.html](https://beam.apache.org/releases/pydoc/2.15.0/_modules/apache_beam/io/gcp/bigquery.html)
 
 I am proofing out a CSV ingest workflow using the Direct Runner and Interactive Runners - the results are the same.  
 

Working in Python 3 (3.7.8 packaged by conda-forge).  

Pip installed via install apache_beam[gcp]

apache_beam.__version__ = 2.23.0

 

Imported from Jira [BEAM-10666](https://issues.apache.org/jira/browse/BEAM-10666). Original Jira may contain additional context.
Reported by: yaquino.

Contributor guide

Open the contributing guide

Research direction

Start with the Python WriteToBigQuery connector and the linked BigQuery documentation for SCHEMA_AUTODETECTION. Reproduce the error using a file-batched workflow with the Direct or Interactive Runner, then verify that schema auto-detection works as documented or that the instructions clearly describe the limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.