Point72 / Point72/csp

Dual dataline optimization

Open
#487 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

tag: internal type: enhancement
Dominant language
Python
Stars
441
Forks
90
Avg merge
1d 12m
Merged PRs (30d)
5

Description

This is an optimization idea to avoid python boxing / unboxing overhead. TBD if there are any issues to actually implement this

The idea is to hold native ( C++ ) and dialect specific ( PyObjectPtr for python ) version of data in the dataline. c++ adapters would only set native data, upon access of the data we would check if dialect specific is valid if not we convert once and update.

Similarly for python outputs, we would only set the dialect value. If a c++ node or c++ adapter needs to access the value, we convert to c++ once if invalid.

Some technical notes:

to be generic for other dialects, and to avoid dialect specific compilation issues, the dialect version would be stored as a raw void *
in order to flag native values as valid or not without introducing a separate bitmask, we can use the low order bit of the void * dialect pointer
for dialects that can read native types natively ( ie c# ) they should be able to avoid accessing dialect side for such types

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the dataline implementation and the C++ adapters and Python outputs described in the issue. Review how native and dialect-specific values are currently stored and converted; done would require a decided generic design for lazy conversion that also accounts for raw void pointers and dialects that can read native values.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, stream-processing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.