GoogleCloudPlatform / GoogleCloudPlatform/PerfKitBenchmarker

PKB beam it_options are not parsed correctly when passing JSON as parameter

Open
#1,548 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2k
Forks
562
Avg merge
4h 55m
Merged PRs (30d)
69

Description

When running HDFS filesystem on beam is necessary to provide --hdfsConfiguration as pipeline option which is JSON format. Example: `--hdfsConfiguration=[{\"fs.defaultFS\":\"hdfs://10.0.0.30:9000\",\"dfs.replication\":1,\"dfs.client.use.datanode.hostname\":\"true\"}]` .
Current parsing logic can't handle that properly as it uses simple split function.
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/4c647df549f8d0d3d0e2d990a8dc6ea383bcbeab/perfkitbenchmarker/beam_pipeline_options.py#L88

If I provide as an input `it_options=["--runner=dataflow", "--hdfsConfiguration=[{\"fs.defaultFS\":\"hdfs://10.0.0.30:9000\",\"dfs.replication\":1}]"]` I will get three elements instead of two.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.