NVIDIA / NVIDIA/cudf

[FEA] No binary column type: Arrow binary/large_binary unsupported, Parquet binary silently becomes string

Open
#23,492 1 comment 0 reactions 0 assignees View on GitHub
feature request libcudf strings
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

## Is your feature request related to a problem? Please describe.

**cuDF has no binary column type.** Arrow `binary` / `large_binary` are neither representable as a cuDF dtype nor convertible from Arrow, and Parquet binary columns are read as **strings**.

Binary payload columns are now standard in multimodal / AI training datasets. Our interleaved schema is:

```python
pa.field("binary_content", pa.large_binary(), nullable=True) # encoded image bytes, ~140 KB/row
pa.field("text_content", pa.string(), nullable=True)
pa.field("embedding", pa.list_(pa.float32()))
```

The payload column has no faithful representation in cuDF.

**What already works, stated up front:** reading Parquet binary through cuDF is *byte-accurate*. Verified by SHA-256 over random bytes, embedded NULs, and empty values; data survives a GPU gather, `.str.byte_count()` returns correct lengths, and a cuDF write-back preserves bytes exactly — on both versions, for both `binary` and `large_binary`. **This is not a data-loss bug.** It is a type-system and interop gap.

### Verified behaviour

Probe output, cuDF 26.08.00a990:

```
--- is there a binary dtype in the cudf namespace? ---
cudf.*binary*/byte* : NONE

--- does cudf accept a binary dtype string? ---
dtype='binary' -> TypeError: data type 'binary' not understood
dtype='large_binary' -> TypeError: data type 'large_binary' not understood
dtype='bytes' -> TypeError: Unsupported type |S0
dtype='|S10' -> TypeError: Unsupported type |S10

--- constructing a Series from python bytes ---
cudf.Series([b"ab", b"cd"])
-> ValueError: CUDF failure at: cpp/src/interop/arrow_utilities.cpp:61:
Unsupported type_id conversion to cudf

--- reader options exposing a binary mode ---
cudf.read_parquet kwargs : no binary-related kwarg
plc.io.parquet.ParquetReaderOptions(Builder): no binary-related option

--- what does cudf call a parquet binary column? ---
parquet says : large_binary
cudf says : str (arrow: large_string)
```

Parquet `binary` and `large_binary` behave identically. What the column is *called* on the way out differs by version, but neither is binary:

| | cuDF dtype after read | Arrow type after read | Parquet type after cuDF write-back |
|---|---|---|---|
| 25.10.00 | `object` | `string` | `string` |
| 26.08.00a990 | `str` | `large_string` | `string` |

Steps/code to reproduce

```python
import cudf, pyarrow as pa, pyarrow.parquet as pq, tempfile, os

# 1. no dtype
for spec in ["binary", "large_binary", "bytes", "|S10"]:
try:
cudf.Series([b"ab"], dtype=spec)
except Exception as e:
print(spec, "->", type(e).__name__, e)

# 2. no arrow interop
try:
cudf.Series([b"ab", b"cd"])
except Exception as e:
print("from bytes ->", type(e).__name__, e)

# 3. type identity lost through parquet (but bytes are fine)
payload = [os.urandom(4096), b"\xff\xd8\x00\xff", b""]
with tempfile.TemporaryDirectory() as d:
src, dst = os.path.join(d, "s.parquet"), os.path.join(d, "d.parquet")
pq.write_table(pa.table({"x": pa.array(payload, type=pa.large_binary())}), src)

df = cudf.read_parquet(src)
print("cudf dtype:", df.dtypes.iloc[0]) # str (26.08) / object (25.10)
arr = df["x"].to_arrow()
print("arrow type:", arr.type) # large_string (26.08) / string (25.10)
got = arr.cast(pa.large_binary()).to_pylist()
print("bytes intact:", got == payload) # True

df.to_parquet(dst, index=False)
print("in :", pq.read_schema(src).field("x").type) # large_binary
print("out:", pq.read_schema(dst).field("x").type) # string <-- identity lost
```

## Describe the solution you'd like

In rough priority order:

1. **Preserve binary type identity through a Parquet round-trip.** Even without a new dtype, a column read from a `binary` / `large_binary` Parquet field should be written back as binary rather than becoming `string`. Smallest fix, largest correctness payoff.
2. **Arrow interop for `binary` / `large_binary`** in `from_arrow` / `to_arrow`. This is what blocks the zero-copy handoff between cuDF and the Arrow-based parts of our pipeline.
3. **A first-class binary dtype**, so payload columns are not conflated with text. Practical consequences of the conflation today: `.to_pylist()` raises `UnicodeDecodeError` on non-UTF-8 payloads (callers must know to `.cast(pa.binary())` first), and every string kernel is nominally applicable to data that is not text.

Exposing the existing `set_convert_binary_to_strings` knob through `pylibcudf` would be a cheaper partial step than (3), though `list` does not round-trip Parquet as binary, so on its own it does not fix (1).

## Describe alternatives you've considered

- **Treat it as a string column** — what we do today. Works for byte fidelity, but loses the type on write and makes the API misleading.
- **`list`** — faithful in memory, but Parquet round-trips it as a LIST, not binary, so it does not interoperate with non-cuDF readers.
- **Keeping payloads out of cuDF entirely** — also what we do today for the multimodal path. Means cuDF cannot participate in payload-bearing tables at all.

## Additional context

- #23493 — the `list` child-element cap. Same multimodal schema, separate root cause; tracked separately.
- rapidsai/cudf#23378 — `ParquetDatasetWriter max_file_size` overestimates list columns (open); relevant to the same multimodal write path.
- The Parquet `FILE` logical type (merged into the spec 2026-07-26) defines an `inline` field of Parquet type `BYTE_ARRAY`, so binary support is a prerequisite for reading that variant of `FILE` columns.

Contributor guide

Open the contributing guide

Research direction

Reproduce the provided Arrow and Parquet examples, then start at cpp/src/interop/arrow_utilities.cpp:61 and trace the from_arrow/to_arrow paths and Parquet read/write behavior. The issue presents several possible scopes; completion should at least preserve binary type identity through a Parquet round-trip, with any Arrow interop or first-class dtype work requiring a separately defined scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.