redpanda-data / redpanda-data/connect

Add discard_unknown to schema_registry_encode

Open
#3,616 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ux
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Currently, schema registry encode processor does not support encoding messages that have extra/unknown fields. The protobuf processor includes a discard_unknown option to discard fields not present in the protobuf definition.

Desired behavior is for the schema_registry_encode processor to match the protobuf processor behavior and expose a discard_unknown configuration so that the below will not result in an error:

Given a proto definition of:

syntax = "proto3";
package testing;

message Person {
  string first_name = 1;
  string last_name = 2;
}

Today, the following message will fail to schema registry encode:

{
"first_name": "Foo",
"middle_name": "Bar",
"last_name": "Baz"
}

with an error similar to "unmarshal 'Person': proto: (line 1:...): unknown field \"middle_name\"",

Tomorrow, ideally this gets encoded without error, resulting in the protobuf messaged published to Redpanda as:

{
"first_name": "Foo",
"last_name": "Baz"
}

Also, we should be able to chain this together to do something like this:

    - schema_registry_encode:
        url: ${SCHEMA_REGISTRY_URL}
        subject: ${! meta("__kafka_topic") }-value
        refresh_period: 10m
    - catch:
        ...
        - schema_registry_encode:
            url: ${SCHEMA_REGISTRY_URL}
            subject: ${! meta("__kafka_topic_with_evolution") }-value
            refresh_period: 10m
            discard_unknown: true

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the schema_registry_encode processor and compare its configuration and behavior with the protobuf processor's discard_unknown option. Add the corresponding configuration behavior, then verify that the Person example encodes while omitting middle_name and that the option works in the shown catch chain.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.