NVIDIA / NVIDIA/cudf

[FEA] Support a Scalar only column

Open
#15,308 4 comments 0 reactions 0 assignees View on GitHub
feature request Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

In an offline discussion with some people in CUDF we expressed how it could be a huge memory savings for the Spark team if we could get columns that we could put into a table/etc but they are really just a scalar with a count. We end up doing this all over the place in all kinds of different situations.

We don't currently use dictionary columns at all but we would be okay with that as an alternative if we could get dictionary columns working more broadly. But even then they still have issues.

In many cases they are more expensive to use for computation if they do work like concat two columns together requires merging the dictionary column instead of a simple memory operation. Yes, a concat of two scalar columns is going to likely be more expensive than two regular columns, but it should not be that bad, and might be faster than generating the fully columns and then concat-ing them.

It also is not always a win from a memory standpoint. With a DICTIONARY32 only values that are on average larger than a 32-bit value result in memory savings, for a scalar column replacement. This can get into really odd cases where an INT32 is not a win from a memory standpoint unless it is null (because the null would add 1 bit per row so 33 bits instead of just 32).

**Describe the solution you'd like**
Ideally https://github.com/rapidsai/cudf/blob/769c1bd6c05f3734044762c9efe3c65ef22cddbd/cpp/include/cudf/column/column_factories.hpp#L546 would just return this new type, or we could have a new API like is used to create a dictionary column from a scalar.

Eventually we might be able to automatically do some things with them, like if we are reading parquet and determine that all of the values in the column are a single thing (like from a dictionary), then we could automatically replace them with a scalar column.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.