Postgresql driver does not support multidimensional arrays
- Dominant language
- C#
- Stars
- 627
- Forks
- 217
- Avg merge
- 22h 22m
- Merged PRs (30d)
- 64
Description
### What happened?
The driver does not seem to support multidimensional arrays:
- Reading multi dimensional array literal flattens them:
```py
from adbc_driver_postgresql.dbapi import connect
url = 'postgresql://scott:tiger@localhost/test'
with connect(url) as conn:
with conn.cursor() as cur:
cur.execute("SELECT '{{a,b},{b,c}}'::text[][] as x")
table = cur.fetch_arrow_table()
print(table)
assert table.to_pydict() == {'x': [[['a', 'b'], ['b', 'c']]]}
```
- Same happens when reading from a table:
```py
from adbc_driver_postgresql.dbapi import connect
url = 'postgresql://scott:tiger@localhost/test'
with connect(url) as conn:
with conn.cursor() as cur:
cur.execute("""CREATE TABLE text_table (text_data text[], text_data2 text[][])""")
cur.execute("INSERT INTO text_table VALUES ('{x,y}', '{{a,b},{c,d}}')")
cur.execute("SELECT text_data, text_data2 FROM text_table")
res = cur.fetchall()
print(res)
assert res == [(["x", "y"], [["a", "b"], ["c", "d"]])], res
```
- insert data into multidimensional array does not work
```py
from adbc_driver_postgresql.dbapi import connect
url = 'postgresql://scott:tiger@localhost/test'
with connect(url) as conn:
with conn.cursor() as cur:
cur.execute("""CREATE TABLE int_table (int_data int[], int_data2 int[][])""")
cur.execute("INSERT INTO int_table VALUES ($1, $2)", ([1,2 ], [[1,2], [2,3]]))
```
Similar erros happen with text types. Passing arrow.scalars does not help either, regardless of splicit typs. Same error with either of these lines:
- `cur.execute("INSERT INTO int_table VALUES ($1, $2)", (pa.scalar([1,2 ]), pa.scalar([[1,2], [2,3]])))`
- `cur.execute("INSERT INTO int_table VALUES ($1, $2)", (pa.scalar([1,2 ], pa.list_(pa.int32())), pa.scalar([[1,2], [2,3]], pa.list_(pa.list_(pa.int32())))))`
have not tried updates, but I guess a similar error would happen.
For my use care the most problematic issue is the fact that dimensionality is lost while reading multidimensional data without any error being reported
### Stack Trace
```
Traceback (most recent call last):
assert table.to_pydict() == {'x': [[['a', 'b'], ['b', 'c']]]}, table.to_pydict()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: {'x': [['a', 'b', 'b', 'c']]}
```
```
Traceback (most recent call last):
assert res == [(["x", "y"], [["a", "b"], ["c", "d"]])], res
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: [(['x', 'y'], ['a', 'b', 'c', 'd'])]
```
```
Traceback (most recent call last):
File "", line 279, in
cur.execute("INSERT INTO int_table VALUES ($1, $2)", ([1,2 ], [[1,2], [2,3]]))
File "C:\Users\csl\miniconda3\envs\mamba\Lib\site-packages\adbc_driver_manager\dbapi.py", line 751, in execute
handle, self._rowcount = _blocking_call(
^^^^^^^^^^^^^^^
File "adbc_driver_manager/_lib.pyx", line 1770, in adbc_driver_manager._lib._blocking_call
return func(*args, **kwargs)
File "adbc_driver_manager/_lib.pyx", line 1364, in adbc_driver_manager._lib.AdbcStatement.execute_query
check_error(status, &c_error)
File "adbc_driver_manager/_lib.pyx", line 261, in adbc_driver_manager._lib.check_error
raise convert_error(status, error)
adbc_driver_manager.InternalError: INTERNAL: nanoarrow call failed: PostgresType::FromSchema(type_resolver, bind_schema->children[i], &type, &na_error) = (22) Invalid argument. Postgres array type with child oid 1016 not found
```
trying with text raises a similar error:
> adbc_driver_manager.InternalError: INTERNAL: nanoarrow call failed: PostgresType::FromSchema(type_resolver, bind_schema->children[i], &type, &na_error) = (22) Invalid argument. Postgres array type with child oid 1009 not found
### How can we reproduce the bug?
See examples above.
### Environment/Setup
adbc_driver_manager = 1.8.0
adbc_driver_postgresql = 1.8.0
Contributor guide
Research direction
Start with the PostgreSQL DB-API paths exercised by connect, cursor.execute, fetch_arrow_table, and fetchall, then inspect the PostgresType::FromSchema error for multidimensional bindings. Done means reads preserve nested array dimensions and inserts accept multidimensional integer and text arrays without the reported errors; reproduce the examples in the issue to verify both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100