[Bug]: beam.io.WriteToBigQuery failed when given schema with space
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
there is any issue in python SDK beam.io.WriteToBigQuery()
when you add a space in schema, like schema="name: STRING", it will fail.
**error message:**
"message": "Invalid value for type: STRING is not a valid value"
**example code:**
`import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
def run():
pipeline_args = []
pipeline_options = PipelineOptions(pipeline_args)
table_ref = 'project_id:dataset_id.table_id'
schema_with_space = "name: STRING"
schema_without_space = "name:STRING"
with beam.Pipeline(options=pipeline_options) as p :
records = p | 'load records' >> beam.Create([{"name":"bob"},{"name":"alice"}])
records | 'write to bigquery' >> beam.io.WriteToBigQuery(
table = table_ref,
schema = schema_with_space,
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED
)
if __name__ == '__main__':
pipeline_args = [
'--runner',
'DirectRunner',
'--project',
'YOUR_PROJECT_ID',
]
run()`
### Issue Priority
Priority: 3 (minor)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [X] Component: Google Cloud Dataflow Runner
Contributor guide
Assessment
This issue has not been assessed yet.