GoogleCloudPlatform / GoogleCloudPlatform/PerfKitBenchmarker
PKB beam it_options are not parsed correctly when passing JSON as parameter
- Dominant language
- Python
- Stars
- 2k
- Forks
- 562
- Avg merge
- 4h 55m
- Merged PRs (30d)
- 69
Description
When running HDFS filesystem on beam is necessary to provide --hdfsConfiguration as pipeline option which is JSON format. Example: `--hdfsConfiguration=[{\"fs.defaultFS\":\"hdfs://10.0.0.30:9000\",\"dfs.replication\":1,\"dfs.client.use.datanode.hostname\":\"true\"}]` .
Current parsing logic can't handle that properly as it uses simple split function.
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/4c647df549f8d0d3d0e2d990a8dc6ea383bcbeab/perfkitbenchmarker/beam_pipeline_options.py#L88
If I provide as an input `it_options=["--runner=dataflow", "--hdfsConfiguration=[{\"fs.defaultFS\":\"hdfs://10.0.0.30:9000\",\"dfs.replication\":1}]"]` I will get three elements instead of two.
Contributor guide
Assessment
This issue has not been assessed yet.