Add fast `csp.Struct` to Arrow record batch conversion
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 441
- Forks
- 90
- Avg merge
- 1d 12m
- Merged PRs (30d)
- 5
Description
Now that @arhamchopra has taken care of fast json conversion, we should move on to fast Apache Arrow conversion. We should have 2 mechanisms for conversion:
- converting an individual struct, necessary but probably less efficient overall
- collecting lists of structs and converting to record batches on either a timer or after a certain window size
Native arrow conversion will make certain use cases substantially faster, including in our csp-based webserver framework, but should also have benefits for other adapters like Kafka and websockets.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the fast JSON conversion work mentioned in the issue and the existing csp.Struct conversion paths. Determine how individual structs and collected lists should become Apache Arrow record batches, including timer or window-size triggers. Done means both conversion mechanisms are implemented and support the stated webserver, Kafka, and websocket use cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100