dssg / dssg/UPSG

Internal UObject masks to prevent unnecessary data copying

Open
#20 0 comments 0 reactions 1 assignee Claimed by @zar1 View on GitHub
enhancement
Dominant language
Python
Stars
9
Forks
6
PR merge metrics
No merged PRs in 30d

Description

There are a number of pipeline stages which don't necessitate copying data, especially those that filter out columns or rows. Rather than making copies, we should have .upsg files with an internal format called "reference" or something like that, which:
1. points to another .upsg file.
2. specifies the filter being applied, so we can populate the full table later

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.