redpanda-data / redpanda-data/redpanda
Pandaproxy: register and validate schema using the schema registry
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 12.6k
- Forks
- 792
- PR merge metrics
- No merged PRs in 30d
Description
Pandaproxy does not provide the ability to validate incoming messages using Avro, Protobuf, or JSON schema stored in the schema registry.
This feature request is to add support for the value_schema and value_schema_id fields so that HTTP clients can post either the full schema or an existing schema ID alongside the records. Pandaproxy shall register the provided schema with the schema registry (in the case of value_schema) or retrieve the existing schema (in the case of value_schema_id) and use the schema to validate and serialise messages before storing them in Redpanda.
For example, this curl command should result in the value_schema schema being registered in the schema registry and used to validate and serialise the list of records before storing them in Redpanda. Pandaproxy should include the value_schema_id in the response:
curl -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" \
--data '{ \
"value_schema": "{\"type\": \"record\", \"name\": \"user\", \"fields\": [{\"name\": \"name\", \"type\": \"string\"}]}", \
"records": [{"value": {"name": "james"}}]}' \
"http://localhost:8082/topics/avrotest"
{"offsets":[{"partition": 0, "offset": 0, "error_code": null, "error": null}], "key_schema_id": null, "value_schema_id": 1}
In subsequent messages only {"value_schema_id": 1, "records":[...]} need be provided and Pandaproxy will fetch the corresponding schema from the schema registry if it isn't cached.
JIRA Link: CORE-1018
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at Pandaproxy's HTTP produce endpoint and its schema registry integration, tracing the requested value_schema and value_schema_id fields. Done means schemas can be registered or retrieved, records are validated and serialized, responses include value_schema_id, and later messages can reuse the schema.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100