duckdb / duckdb/duckdb-python

Arrow w/ arrow_output_version produces errors with VARINT, GEOMETRY and ENUMs

Offen
#517 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
needs triage
Vorherrschende Sprache
Python
Sterne
187
Forks
112
Ø Merge
13 Std. 29 Min.
Gemergte PRs (30 T.)
17

Beschreibung

### What happens?

When using arrow_output_version (documented here: https://duckdb.org/docs/current/configuration/overview), certain data types fail with a "Expected 3 buffers" error.

See below reproducer.

Output:

> duckdb 1.5.4 | pyarrow 24.0.0
>
> VARINT defaults (sv=false, aov=1.0) : OK
> VARINT aov=1.4 : FAIL -> Expected 3 buffers for imported type extension, ArrowArray struct has 4
> GEOMETRY aov=1.5 : FAIL -> Expected 3 buffers for imported type binary, ArrowArray struct has 4
> ENUM sv=true + aov=1.5 : FAIL -> Expected 3 buffers for imported type string, ArrowArray struct has 4

### To Reproduce

```py
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.12"
# dependencies = [
# "duckdb==1.5.4",
# "pyarrow>=22",
# ]
# ///

import duckdb
import pyarrow as pa

print(f"duckdb {duckdb.__version__} | pyarrow {pa.__version__}\n")

# VARINT/BIGNUM: defaults are fine
con = duckdb.connect()
pa.table(con.sql("SELECT (2**100)::VARINT AS v"))
print("VARINT defaults (sv=false, aov=1.0): OK")

# VARINT/BIGNUM: arrow_output_version >= 1.4 -> FAIL
con = duckdb.connect()
con.execute("SET arrow_output_version='1.4'")
try:
pa.table(con.sql("SELECT (2**100)::VARINT AS v"))
print("VARINT aov=1.4: OK (unexpected)")
except Exception as e:
print(f"VARINT aov=1.4: FAIL -> {e}")

# --- GEOMETRY: arrow_output_version = 1.5 -> FAIL
con = duckdb.connect()
con.execute("INSTALL spatial")
con.execute("LOAD spatial")
con.execute("SET arrow_output_version='1.5'")
try:
pa.table(con.sql("SELECT ST_Point(1, 2) AS g"))
print("GEOMETRY aov=1.5: OK (unexpected)")
except Exception as e:
print(f"GEOMETRY aov=1.5: FAIL -> {e}")

# --- ENUM: needs BOTH produce_arrow_string_view=true AND aov>=1.5 -----------
con = duckdb.connect()
con.execute("SET produce_arrow_string_view=true")
con.execute("SET arrow_output_version='1.5'")
try:
pa.table(con.sql("SELECT 'happy'::ENUM('happy', 'sad') AS v"))
print("ENUM sv=true + aov=1.5: OK (unexpected)")
except Exception as e:
print(f"ENUM sv=true + aov=1.5: FAIL -> {e}")

```

### OS:

Linux

### DuckDB Package Version:

1.5.4

### Python Version:

3.14

### Full Name:

paultiq

### Affiliation:

Iqmo

### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

### Did you include all relevant data sets for reproducing the issue?

Yes

### Did you include all code required to reproduce the issue?

- [x] Yes, I have

### Did you include all relevant configuration to reproduce the issue?

- [x] Yes, I have

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne damit, den im Issue enthaltenen Python-Reproducer mit DuckDB 1.5.4 und PyArrow auszuführen, und verfolge dann die Arrow-Konvertierungspfade, die von VARINT, GEOMETRY und ENUM mit aktiviertem arrow_output_version durchlaufen werden. Als erledigt gilt die Aufgabe, wenn alle drei Konvertierungen PyArrow-Tabellen ohne den Fehler „Expected 3 buffers“ erzeugen, während das gemeldete Standardverhalten erhalten bleibt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, sql
Bereich
data
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
48/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.