Avro deserialization
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 10
- Forks
- 1
- Avg merge
- 10m
- Merged PRs (30d)
- 16
Description
Hi!
I'm looking for an efficient way to stream data from Apache Kafka to Ducklake and millpond seems perfect.
The only limit I see is the reliance on JSON as single serialization format. I think Avro makes sense for high throughput data.
Your documentation says: "If you need Avro/Protobuf deserialization with schema registry, you handle it yourself.".
I was wondering if that means that you do not wish to add Avro support or whether you would be open to contributions.
Kind regards
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating the Kafka ingestion path and the existing JSON deserialization flow, then determine how schema-registry handling would fit the pipeline. Done would require an agreed Avro support scope and validation that Avro data can be streamed into Ducklake.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, python
- Domain
- data-engineering, databases, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100