redpanda-data / redpanda-data/redpanda

Pandaproxy: register and validate schema using the schema registry

Open
#6,370 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/pandaproxy area/schema-registry kind/enhance
Dominant language
C++
Stars
12.6k
Forks
792
PR merge metrics
No merged PRs in 30d

Description

Pandaproxy does not provide the ability to validate incoming messages using Avro, Protobuf, or JSON schema stored in the schema registry.

This feature request is to add support for the value_schema and value_schema_id fields so that HTTP clients can post either the full schema or an existing schema ID alongside the records. Pandaproxy shall register the provided schema with the schema registry (in the case of value_schema) or retrieve the existing schema (in the case of value_schema_id) and use the schema to validate and serialise messages before storing them in Redpanda.

For example, this curl command should result in the value_schema schema being registered in the schema registry and used to validate and serialise the list of records before storing them in Redpanda. Pandaproxy should include the value_schema_id in the response:

curl -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" \
     --data '{ \
         "value_schema": "{\"type\": \"record\", \"name\": \"user\", \"fields\": [{\"name\": \"name\", \"type\": \"string\"}]}", \ 
         "records": [{"value": {"name": "james"}}]}' \
     "http://localhost:8082/topics/avrotest"

{"offsets":[{"partition": 0, "offset": 0, "error_code": null, "error": null}], "key_schema_id": null, "value_schema_id": 1}

In subsequent messages only {"value_schema_id": 1, "records":[...]} need be provided and Pandaproxy will fetch the corresponding schema from the schema registry if it isn't cached.

JIRA Link: CORE-1018

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at Pandaproxy's HTTP produce endpoint and its schema registry integration, tracing the requested value_schema and value_schema_id fields. Done means schemas can be registered or retrieved, records are validated and serialized, responses include value_schema_id, and later messages can reuse the schema.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.