apache / apache/beam

[Bug]: beam.io.WriteToBigQuery failed when given schema with space

Open
#25,704 3 comments 0 reactions 0 assignees View on GitHub
bigquery bug dataflow P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

there is any issue in python SDK beam.io.WriteToBigQuery()
when you add a space in schema, like schema="name: STRING", it will fail.

**error message:**
"message": "Invalid value for type: STRING is not a valid value"

**example code:**

`import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions

def run():
pipeline_args = []
pipeline_options = PipelineOptions(pipeline_args)

table_ref = 'project_id:dataset_id.table_id'
schema_with_space = "name: STRING"
schema_without_space = "name:STRING"

with beam.Pipeline(options=pipeline_options) as p :
records = p | 'load records' >> beam.Create([{"name":"bob"},{"name":"alice"}])
records | 'write to bigquery' >> beam.io.WriteToBigQuery(
table = table_ref,
schema = schema_with_space,
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED
)

if __name__ == '__main__':
pipeline_args = [
'--runner',
'DirectRunner',
'--project',
'YOUR_PROJECT_ID',
]

run()`

### Issue Priority

Priority: 3 (minor)

### Issue Components

- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [X] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.