Writing encoded flows to Kafka
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Lua
- Stars
- 1.8k
- Forks
- 51
- PR merge metrics
- No merged PRs in 30d
Description
When writing thousands of complex json flows/second to kafka, we find that the template keys are taking a lot of disk space. Would it be possible to use some kind of encoding (Avro, Ndpi TLV ...) in order to have more compact data?
Thanks
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue provides no repository file, entry point, or test to start from. First inspect how complex JSON flows are serialized and written to Kafka, then evaluate the requested compact encoding options. Done would require a defined encoding approach and evidence that it reduces storage for the stated flow volume.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- json, kafka
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100