DALI Numba Plugin
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 5.8k
- Forks
- 678
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 27
Description
I've hacked together a quick prototype which demonstrates how you could use numba cfunc to process data.
numba is a jit to native compiler for python. The cfunc feature allows you to compile a function, get a pointer, and share that pointer with c/c++.
Here's an overview:
- define the function parameter types
- define the function
- provide the function pointer to the op
- call the function in the operator
Change the function definition and see how min/max/std changes.
I plan on developing this further, but I'm curious if this would be something you'd be interested in merging into the DALI repo when it was more mature.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked gartnera/dali-numba-plugin prototype, especially demo.py lines 6, 10, and 34 and dummy.cc line 19, to understand the proposed function-pointer flow. Compare the prototype with DALI's existing operator and plugin interfaces, then determine the integration scope and acceptance criteria before implementing anything.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100