dolthub / dolthub/doltpy

Use SQL data types when reading into pandas

Open
#179 6 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
62
Forks
12
PR merge metrics
No merged PRs in 30d

Description

This is a feature request. Currently, `read_pandas` and `read_pandas_sql` use SQL->CSV->Pandas conversion [behind the scenes](https://github.com/dolthub/doltpy/blob/f3c83cc7f686d0e65f35c9fc32dc567519a862c3/doltpy/cli/read.py#L20). It would be great if the resulting DataFrame could inherit some info from the SQL table schema, such as data types.

Minimal example:

```sql
dolt_database> CREATE TABLE mytable (
-> id int NOT NULL,
-> is_human bool(1),
-> PRIMARY KEY (id)
-> );
dolt_database> INSERT INTO mytable
-> VALUES (1, 1);
Query OK, 1 row affected
dolt_database> INSERT INTO mytable VALUES (2, 0);
Query OK, 1 row affected
dolt_database> SELECT * FROM mytable;
+----+----------+
| id | is_human |
+----+----------+
| 1 | 1 |
| 2 | 0 |
+----+----------+
```

```python
In [5]: read_pandas_sql(dolt, "SELECT * from mytable")
Out[5]:
id is_human
0 1 1
1 2 0

In [7]: df.dtypes
Out[7]:
id object
is_human object
dtype: object
```

If this feature is implemented, the result would be:

```python
In [11]: df.dtypes
Out[11]:
id int64
is_human bool
dtype: object
```

I appreciate that this is a time consuming change. I believe though that many use cases would benefit. In my case, I often use "read table via dolt, update dataframe in pandas, write updated table via dolt" workflow. Having more intelligent data type resolution would help me avoid different dtypes within a column. It would also reduce queries like:

```python
df[df["my_bool_column"] == "1"]
```

Instead, I'd use
```python
df[df["my_bool_column"]]
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in doltpy/cli/read.py at the read_pandas and read_pandas_sql entry points, then trace the SQL-to-CSV-to-Pandas conversion described in the issue. Compare the SQL schema with the resulting DataFrame dtypes for the minimal example, and verify that the expected integer and boolean types are preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python, sql
Domain
data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.