Dual dataline optimization
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 441
- Forks
- 90
- Avg merge
- 1d 12m
- Merged PRs (30d)
- 5
Description
This is an optimization idea to avoid python boxing / unboxing overhead. TBD if there are any issues to actually implement this
The idea is to hold native ( C++ ) and dialect specific ( PyObjectPtr for python ) version of data in the dataline. c++ adapters would only set native data, upon access of the data we would check if dialect specific is valid if not we convert once and update.
Similarly for python outputs, we would only set the dialect value. If a c++ node or c++ adapter needs to access the value, we convert to c++ once if invalid.
Some technical notes:
to be generic for other dialects, and to avoid dialect specific compilation issues, the dialect version would be stored as a raw void *
in order to flag native values as valid or not without introducing a separate bitmask, we can use the low order bit of the void * dialect pointer
for dialects that can read native types natively ( ie c# ) they should be able to avoid accessing dialect side for such types
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the dataline implementation and the C++ adapters and Python outputs described in the issue. Review how native and dialect-specific values are currently stored and converted; done would require a decided generic design for lazy conversion that also accounts for raw void pointers and dialects that can read native values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend, stream-processing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100