apache / apache/beam

TriggerLoadJobs.process in bigquery_file_loads schema is type str

Open
#19,950 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

 I've found a first issue with the BigQueryFileLoads Transform and the type of the schema parameter.
```

Triggering job beam_load_2019_10_11_140829_19_157670e4d458f0ff578fbe971a91b30a_1570802915 to load data
to BigQuery table .Schema: {"fields": [{"name": "id", "type": "INTEGER", "mode": "required"}, {"name":
"description", "type": "STRING", "mode": "nullable"}]}. Additional parameters: {}
Retry with exponential
backoff: waiting for 4.875033410381894 seconds before retrying _insert_load_job because we caught exception:
apitools.base.protorpclite.messages.ValidationError: Expected type
for field schema, found {"fields": [{"name": "id", "type": "INTEGER", "mode": "required"}, {"name":
"description", "type"
: "STRING", "mode": "nullable"}]} (type )
Traceback for above
exception (most recent call last):
 File "/opt/conda/lib/python3.7/site-packages/apache_beam/utils/retry.py",
line 206, in wrapper
   return fun(*args, **kwargs)
 File "/opt/conda/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery_tools.py",
line 344, in _insert_load_job
   **additional_load_parameters
 File "/opt/conda/lib/python3.7/site-packages/apitools/base/protorpclite/messages.py",
line 791, in __init__
   setattr(self, name, value)
 File "/opt/conda/lib/python3.7/site-packages/apitools/base/protorpclite/messages.py",
line 973, in __setattr__
   object.__setattr__(self, name, value)
 File "/opt/conda/lib/python3.7/site-packages/apitools/base/protorpclite/messages.py",
line 1652, in __set__
   super(MessageField, self).__set__(message_instance, value)
 File "/opt/conda/lib/python3.7/site-packages/apitools/base/protorpclite/messages.py",
line 1293, in __set__
   value = self.validate(value)
 File "/opt/conda/lib/python3.7/site-packages/apitools/base/protorpclite/messages.py",
line 1400, in validate
   return self.__validate(value, self.validate_element)
 File "/opt/conda/lib/python3.7/site-packages/apitools/base/protorpclite/messages.py",
line 1358, in __validate
   return validate_element(value)  
 File "/opt/conda/lib/python3.7/site-packages/apitools/base/protorpclite/messages.py",
line 1340, in validate_element
   (self.type, name, value, type(value)))
 

```

 

The triggering code looks like this:

 
options.view_as(DebugOptions).experiments = ['use_beam_bq_sink']
       # Save main session state so pickled functions and classes
       # defined in __main__ can be unpickled
       options.view_as(SetupOptions).save_main_session = True
       custom_options = options.view_as(LoadSqlToBqOptions)
       with beam.Pipeline(options=options) as p:
           (p
               | "Initializing with empty collection" \>\> beam.Create([1])
               | "Reading records from CloudSql" \>\> beam.ParDo(ReadFromRelationalDBFn(
                   username=custom_options.user,
                   password=custom_options.password,
                   database=custom_options.database,
                   table=custom_options.table,
                   key_field=custom_options.key_field,
                   batch_size=custom_options.batch_size))
               | "Converting Row Object for BigQuery" \>\> beam.ParDo(BuildForBigQueryFn(custom_options.bq_schema))
               | "Writing to BigQuery" \>\> beam.io.WriteToBigQuery(
                       table=custom_options.bq_table,
                       schema=custom_options.bq_schema,
                       write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE,
                       create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED))
 

 

Imported from Jira [BEAM-8452](https://issues.apache.org/jira/browse/BEAM-8452). Original Jira may contain additional context.
Reported by: ngoodrich.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.