GoogleCloudPlatform / GoogleCloudPlatform/professional-services

[dataflow-data-generator] TypeError: '>' not supported between instances of 'str' and 'int' [while running 'Generate Data']

Open
#460 3 comments 0 reactions 1 assignee Claimed by @jaketf View on GitHub
Dominant language
Python
Stars
3.1k
Forks
1.5k
Avg merge
2d 8h
Merged PRs (30d)
44

Description

Looks like there is an issue in the code that is comparing a string with an int:

record[fieldname] > max_size:

Looks like record[fieldname] holds a str instead of an int

**Running this command:**
python3 data-generator-pipeline/data_generator_pipeline.py \
--project=bank-bigquery --setup_file=data-generator-pipeline/setup.py \
--worker_machine_type=n1-highcpu-32 --runner=DataflowRunner \
--staging_location=gs://bank-raw-data/test --temp_location=gs://bank-raw-data/temp \
--schema_file=gs://bank-raw-data/schemas/DMR1999_AU_CONTRATOS.json --output_bq_table=datamart.DMR1999_AU_CONTRATOS \
--p_null=0.05 \
--num_records=10000 --save_main_session

**I got this error:**
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 647, in do_work
work_executor.execute()
File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 176, in execute
op.start()
File "dataflow_worker/native_operations.py", line 38, in dataflow_worker.native_operations.NativeReadOperation.start
File "dataflow_worker/native_operations.py", line 39, in dataflow_worker.native_operations.NativeReadOperation.start
File "dataflow_worker/native_operations.py", line 44, in dataflow_worker.native_operations.NativeReadOperation.start
File "dataflow_worker/native_operations.py", line 54, in dataflow_worker.native_operations.NativeReadOperation.start
File "apache_beam/runners/worker/operations.py", line 329, in apache_beam.runners.worker.operations.Operation.output
File "apache_beam/runners/worker/operations.py", line 192, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 682, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 683, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 952, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1028, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "/usr/local/lib/python3.7/site-packages/future/utils/__init__.py", line 421, in raise_with_traceback
raise exc.with_traceback(traceback)
File "apache_beam/runners/common.py", line 950, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 547, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam/runners/common.py", line 1078, in apache_beam.runners.common._OutputProcessor.process_outputs
File "/usr/local/lib/python3.7/site-packages/data_generator/PrettyDataGenerator.py", line 582, in process
row = self.generate_fake(fschema=faker_schema, key_dict=element)
File "/usr/local/lib/python3.7/site-packages/data_generator/PrettyDataGenerator.py", line 550, in generate_fake
data = self.sanity_check(data, col_name)
File "/usr/local/lib/python3.7/site-packages/data_generator/PrettyDataGenerator.py", line 415, in sanity_check
if record[fieldname] > max_size:
TypeError: '>' not supported between instances of 'str' and 'int' [while running 'Generate Data']

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.