redpanda-data / redpanda-data/connect
Add discard_unknown to schema_registry_encode
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
Currently, schema registry encode processor does not support encoding messages that have extra/unknown fields. The protobuf processor includes a discard_unknown option to discard fields not present in the protobuf definition.
Desired behavior is for the schema_registry_encode processor to match the protobuf processor behavior and expose a discard_unknown configuration so that the below will not result in an error:
Given a proto definition of:
syntax = "proto3";
package testing;
message Person {
string first_name = 1;
string last_name = 2;
}
Today, the following message will fail to schema registry encode:
{
"first_name": "Foo",
"middle_name": "Bar",
"last_name": "Baz"
}
with an error similar to "unmarshal 'Person': proto: (line 1:...): unknown field \"middle_name\"",
Tomorrow, ideally this gets encoded without error, resulting in the protobuf messaged published to Redpanda as:
{
"first_name": "Foo",
"last_name": "Baz"
}
Also, we should be able to chain this together to do something like this:
- schema_registry_encode:
url: ${SCHEMA_REGISTRY_URL}
subject: ${! meta("__kafka_topic") }-value
refresh_period: 10m
- catch:
...
- schema_registry_encode:
url: ${SCHEMA_REGISTRY_URL}
subject: ${! meta("__kafka_topic_with_evolution") }-value
refresh_period: 10m
discard_unknown: true
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the schema_registry_encode processor and compare its configuration and behavior with the protobuf processor's discard_unknown option. Add the corresponding configuration behavior, then verify that the Person example encodes while omitting middle_name and that the option works in the shown catch chain.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100