redpanda-data / redpanda-data/connect
Iceberg output: the `row_operation` example uses a non-existent bloblang ternary
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
The third published example for the iceberg output's row_operation field cannot
be used as written. It appears in the field's Examples block on the docs site and
in the in-repo generated docs:
row_operation: ${! this.op == "d" ? "delete" : "upsert" }
1. Bloblang has no ternary operator. Even correctly quoted, the mapping does not
parse:
$ echo '{"op":"d"}' | rpk connect blobl 'root = this.op == "d" ? "delete" : "upsert"'
failed to parse mapping: line 1 char 23: expected line break
|
1 | root = this.op == "d" ? "delete" : "upsert"
| ^---
Inside a config, as an interpolated string, the same expression is rejected at lint
time:
$ rpk connect lint repro.yaml
repro.yaml(8,40) required: expected end of expression, got: ? "de
2. As published, unquoted, it is not valid YAML. The : inside the value makes
the parser read it as a nested mapping, so the config fails before Bloblang is even
reached:
$ rpk connect lint repro.yaml
repro.yaml(1,1) yaml: line 8: mapping values are not allowed in this context
Reproduction
repro.yaml, using the example verbatim:
input:
generate: { count: 1, mapping: 'root.op = "d"', interval: "" }
output:
iceberg:
catalog: { url: "http://127.0.0.1:8181" }
namespace: n
table: t
row_operation: ${! this.op == "d" ? "delete" : "upsert" }
identifier_fields: [ id ]
max_in_flight: 1
storage: { aws_s3: { bucket: b, region: us-east-1 } }
rpk connect lint repro.yaml fails with the YAML error above. Quoting the value
changes it to the Bloblang parse error above. Neither form runs.
Reproduced on Redpanda Connect 4.102.0 and 4.105.0.
Where it lives
- Source of truth:
internal/impl/iceberg/config.go, therow_operationfield's
thirdExample(...)call (line ~322 onmainat time of writing). - Generated doc in-repo:
docs/modules/components/pages/outputs/iceberg.adoc. - Published pages showing it:
- https://docs.redpanda.com/connect/components/outputs/iceberg/#row_operation
- the Redpanda Cloud mirror of the same component page
The other two examples on that field, insert and ${! metadata("op") }, are both
correct, as is the row_operation line in the change-data-capture example further
down the page.
Suggested fix
Replace the ternary example with either of these, both of which lint and run:
row_operation: '${! match this.op { "d" => "delete", _ => "upsert" } }'
row_operation: '${! if this.op == "d" { "delete" } else { "upsert" } }'
i.e. in config.go:
Example(`${! match this.op { "d" => "delete", _ => "upsert" } }`).
then regenerate the in-repo docs with make docs (cmd/tools/docs_gen).
Quoting the value in the rendered example is worth considering as a general habit
for interpolated fields, since any expression containing a : hits the same YAML
ambiguity even when the Bloblang itself is valid.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in internal/impl/iceberg/config.go at the row_operation field's third Example call. Replace the invalid ternary example with the suggested valid expression, run make docs, and inspect docs/modules/components/pages/outputs/iceberg.adoc. Done means the source and generated examples show a Bloblang expression that parses and avoids the published YAML ambiguity.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100