duckdb / duckdb/duckdb-python

Crashes while handling non-select result set (DataFrame)

Ouverte
#428 0 commentaires 1 réaction 0 personnes assignées Voir sur GitHub
needs triage
Langage dominant
Python
Étoiles
187
Forks
112
Merge moyen
13 h 29 min
PR mergées (30 j)
17

Description

### What happens?

Hi.

### Problem

As the result of a `sparl.sql("non-select")` where `non-select` is any SQL statement that is not a select, e.g., USE, INSERT, DROP, CREATE, ... the `sql()` function will correctly return an empty DataFrame, which is the behavior of the pyspark API.

However, that object crashes when using any of its APIs, because the internal `relation` object is None. The same applies when trying to create an empty DataFrame without columns. A

### Fix

I think the best fix would require fixing the underlying c++ Relation object from the duckdb C++ library to support an empty relation without columns. There are also a couple other fixes like allowing the underlying `duckdb.struct_type()` to have no fields. That would make the low-level API more robust and require less patching in the python layer.

Then the `DuckDBPyConnection::RunQuery` function needs to return an empty relation for non-select statement, instead of `nullptr`. All these fixes felt a bit overwhelming so I won't submit a patch.

### To Reproduce

Testcase. All this works with Spark.
```
@pytest.mark.parametrize("mode", ["pandas", "list", "non-select"])
def test_empty_sdf( spark_session_g, mode):
from pyspark.sql import functions as f
from pyspark.sql import types as t
import pandas as pd

spark = spark_session_g
if mode =="pandas":
sdf = spark.createDataFrame(pd.DataFrame(), t.StructType([]))
elif mode == "list":
sdf = spark.createDataFrame([], t.StructType([]))
else:
curr_db = spark.catalog.currentDatabase()
sdf = spark.sql(f"USE {curr_db}") # non-result set query

assert sdf.schema == t.StructType([])
assert sdf.columns == []
assert sdf.collect() == []
assert sdf.toPandas().empty
assert sdf.toArrow().shape == (0, 0)
sdf.createOrReplaceTempView("my_vv1")
assert spark.sql("SELECT * from my_vv1").toArrow().shape == (0, 0)
sdf.show() # no-op, no crash
assert sdf.withColumn("col1", f.lit(1)).columns == ["col1"]
assert sdf.withColumns({"col1": f.lit(1)}).columns == ["col1"]
assert sdf.drop("noop").columns == []
```

### OS:

Any

### DuckDB Package Version:

Main branch

### Python Version:

3.12

### Full Name:

João Eiras

### Affiliation:

private

### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a source build

### Did you include all relevant data sets for reproducing the issue?

Yes

### Did you include all code required to reproduce the issue?

- [x] Yes, I have

### Did you include all relevant configuration to reproduce the issue?

- [x] Yes, I have

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Commencez par le cas de test inline pour les DataFrames vides et inspectez DuckDBPyConnection::RunQuery, l’objet Relation en C++ et duckdb.struct_type(). Le travail est terminé lorsque le SQL qui n’est pas une sélection et les DataFrames vides sans colonnes prennent en charge les API DataFrame indiquées sans provoquer de plantage, notamment le schéma, la collection, la conversion Arrow, les vues temporaires et les opérations sur les colonnes.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python, sql
Domaine
api, databases
Type d'issue
Bug
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
Calme
Clarté
Plutôt claire
Accessibilité débutants
35/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.