apache / apache/datasketches-python

Vectorize update()

Open
#21 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
46
Forks
9
PR merge metrics
No merged PRs in 30d

Description

Looping in python is slow. We should enable `update()` with multiple inputs where C++ handles the iteration, for all sketches.

For sketches that take primitive types this is simple and can be done by overloading `update()`. For item containers it may be less straightforward since a list is a type of object, meaning an overloaded may treat the list as a single thing to ingest.

IIRC, the wrappers (both pybind11 and nanobind) iterate through the possible methods in the order ini which they were declared in the wrapper definition, which is not good for API design: Lack of transparency, internal rearranging of code can cause side-effects, etc. So we probably need a different method name. Then we run into the question of whether we should use an overload where practical and a different name where necessary or if we go for more consistency.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the existing update() entry points and the pybind11 and nanobind wrapper definitions for each sketch. Determine how primitive inputs and item containers are distinguished, then settle whether vectorization uses overloads or a separate method name. Done means all sketches support C++-side iteration with a consistent, transparent Python API.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
api, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.