NVIDIA-Merlin / NVIDIA-Merlin/Merlin
[INF] Merlin Commons
Open
@EvenOldridge is already working on this.
Since Dec 21, 2022.
infrastructure
roadmap
- Dominant language
- Python
- Stars
- 907
- Forks
- 129
- PR merge metrics
- No merged PRs in 30d
Description
Problem:
Merlin now has a bunch of libraries that need to interoperate smoothly, but a general lack of shared abstractions, conventions, and standards that would make that possible.
Goal:
- Build a solid foundation for the Merlin libraries via improvements in Core
New Functionality
- Core:
- Shape in column schemas (for consistent tracking across libraries)
- Cross-framework dtype translation (e.g. via Merlin dtypes)
- Cross-framework data transfer via zero-copy protocols (for Columns and DictArrays -> Series and Dataframes)
- Bespoke Merlin schema file format (i.e. a Protobuf schema for Merlin schema that isn't from Tensorflow Metadata)
- Corresponding updates in all downstream libraries
Constraints:
- All functionality entailed by this issue has to work in and be adoptable by all Merlin libraries
Starting Point:
- Proposals:
- Shapes - https://docs.google.com/document/d/13WrHuCxspTDjv2NPQxWtzZqBoI75Np5_tOopJEWQbBg/edit#
- Dtypes
- Data transfer
- Schema file format
- #791
- Overarching decisions to be made (Merlin commons ) https://docs.google.com/document/d/1IGCS5-f6-SJkpfHFChE4L_iaNRXVjrx0zW-NPDOyfkU/edit
- #813
- #816
- #814
- #815
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.