GoogleCloudPlatform / GoogleCloudPlatform/professional-services
[dataflow-data-generator] TypeError: '>' not supported between instances of 'str' and 'int' [while running 'Generate Data']
- Dominant language
- Python
- Stars
- 3.1k
- Forks
- 1.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 44
Description
Looks like there is an issue in the code that is comparing a string with an int:
record[fieldname] > max_size:
Looks like record[fieldname] holds a str instead of an int
**Running this command:**
python3 data-generator-pipeline/data_generator_pipeline.py \
--project=bank-bigquery --setup_file=data-generator-pipeline/setup.py \
--worker_machine_type=n1-highcpu-32 --runner=DataflowRunner \
--staging_location=gs://bank-raw-data/test --temp_location=gs://bank-raw-data/temp \
--schema_file=gs://bank-raw-data/schemas/DMR1999_AU_CONTRATOS.json --output_bq_table=datamart.DMR1999_AU_CONTRATOS \
--p_null=0.05 \
--num_records=10000 --save_main_session
**I got this error:**
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 647, in do_work
work_executor.execute()
File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 176, in execute
op.start()
File "dataflow_worker/native_operations.py", line 38, in dataflow_worker.native_operations.NativeReadOperation.start
File "dataflow_worker/native_operations.py", line 39, in dataflow_worker.native_operations.NativeReadOperation.start
File "dataflow_worker/native_operations.py", line 44, in dataflow_worker.native_operations.NativeReadOperation.start
File "dataflow_worker/native_operations.py", line 54, in dataflow_worker.native_operations.NativeReadOperation.start
File "apache_beam/runners/worker/operations.py", line 329, in apache_beam.runners.worker.operations.Operation.output
File "apache_beam/runners/worker/operations.py", line 192, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 682, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 683, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 952, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1028, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "/usr/local/lib/python3.7/site-packages/future/utils/__init__.py", line 421, in raise_with_traceback
raise exc.with_traceback(traceback)
File "apache_beam/runners/common.py", line 950, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 547, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam/runners/common.py", line 1078, in apache_beam.runners.common._OutputProcessor.process_outputs
File "/usr/local/lib/python3.7/site-packages/data_generator/PrettyDataGenerator.py", line 582, in process
row = self.generate_fake(fschema=faker_schema, key_dict=element)
File "/usr/local/lib/python3.7/site-packages/data_generator/PrettyDataGenerator.py", line 550, in generate_fake
data = self.sanity_check(data, col_name)
File "/usr/local/lib/python3.7/site-packages/data_generator/PrettyDataGenerator.py", line 415, in sanity_check
if record[fieldname] > max_size:
TypeError: '>' not supported between instances of 'str' and 'int' [while running 'Generate Data']
Contributor guide
Assessment
This issue has not been assessed yet.