redpanda-data / redpanda-data/connect

Iceberg output: the `row_operation` example uses a non-existent bloblang ternary

Open Beginner friendly
#4,711 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

The third published example for the iceberg output's row_operation field cannot
be used as written. It appears in the field's Examples block on the docs site and
in the in-repo generated docs:

row_operation: ${! this.op == "d" ? "delete" : "upsert" }

1. Bloblang has no ternary operator. Even correctly quoted, the mapping does not
parse:

$ echo '{"op":"d"}' | rpk connect blobl 'root = this.op == "d" ? "delete" : "upsert"'
failed to parse mapping: line 1 char 23: expected line break
  |
1 | root = this.op == "d" ? "delete" : "upsert"
  |                       ^---

Inside a config, as an interpolated string, the same expression is rejected at lint
time:

$ rpk connect lint repro.yaml
repro.yaml(8,40) required: expected end of expression, got: ? "de

2. As published, unquoted, it is not valid YAML. The : inside the value makes
the parser read it as a nested mapping, so the config fails before Bloblang is even
reached:

$ rpk connect lint repro.yaml
repro.yaml(1,1) yaml: line 8: mapping values are not allowed in this context
Reproduction

repro.yaml, using the example verbatim:

input:
  generate: { count: 1, mapping: 'root.op = "d"', interval: "" }
output:
  iceberg:
    catalog: { url: "http://127.0.0.1:8181" }
    namespace: n
    table: t
    row_operation: ${! this.op == "d" ? "delete" : "upsert" }
    identifier_fields: [ id ]
    max_in_flight: 1
    storage: { aws_s3: { bucket: b, region: us-east-1 } }

rpk connect lint repro.yaml fails with the YAML error above. Quoting the value
changes it to the Bloblang parse error above. Neither form runs.

Reproduced on Redpanda Connect 4.102.0 and 4.105.0.

Where it lives

The other two examples on that field, insert and ${! metadata("op") }, are both
correct, as is the row_operation line in the change-data-capture example further
down the page.

Suggested fix

Replace the ternary example with either of these, both of which lint and run:

row_operation: '${! match this.op { "d" => "delete", _ => "upsert" } }'
row_operation: '${! if this.op == "d" { "delete" } else { "upsert" } }'

i.e. in config.go:

Example(`${! match this.op { "d" => "delete", _ => "upsert" } }`).

then regenerate the in-repo docs with make docs (cmd/tools/docs_gen).

Quoting the value in the rendered example is worth considering as a general habit
for interpolated fields, since any expression containing a : hits the same YAML
ambiguity even when the Bloblang itself is valid.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in internal/impl/iceberg/config.go at the row_operation field's third Example call. Replace the invalid ternary example with the suggested valid expression, run make docs, and inspect docs/modules/components/pages/outputs/iceberg.adoc. Done means the source and generated examples show a Bloblang expression that parses and avoids the published YAML ambiguity.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.