dlt-hub / dlt-hub/dlt

[Backend - Pyarrow]PostgresSQL ingestion maps ARRAY to wrong datatype in Bigquery

Open
#2,137 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### dlt version

1.4.1

### Describe the problem

When ingesting ARRAY bigquery the pyarrow wrongly maps to a RECORD inside a RECORD. I remember seeing this before and it's a pyarrow problem

### Steps to reproduce

# How to replicate

Create a .env

```
EXTRACT_DB_DELTA__DESTINATION__BIGQUERY__LOCATION
EXTRACT_DB_DELTA__DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID
EXTRACT_DB_DELTA__SOURCES__SQL_DATABASE__SQL_DATABASE__CREDENTIALS__DRIVERNAME
EXTRACT_DB_DELTA__SOURCES__SQL_DATABASE__SQL_DATABASE__CREDENTIALS__HOST
EXTRACT_DB_DELTA__SOURCES__SQL_DATABASE__SQL_DATABASE__CREDENTIALS__USERNAME
EXTRACT_DB_DELTA__SOURCES__SQL_DATABASE__SQL_DATABASE__CREDENTIALS__PASSWORD
EXTRACT_DB_DELTA__SOURCES__SQL_DATABASE__SQL_DATABASE__CREDENTIALS__DATABASE
EXTRACT_DB_DELTA__SOURCES__SQL_DATABASE__SQL_DATABASE__CREDENTIALS__PORT
```

Docker-compose.yml

```yaml
version: '3.8'
services:
db:
image: postgres:15-alpine
environment:
POSTGRES_USER: ${EXTRACT_DB_DELTA__SOURCES__SQL_DATABASE__SQL_DATABASE__CREDENTIALS__USERNAME}
POSTGRES_PASSWORD: ${EXTRACT_DB_DELTA__SOURCES__SQL_DATABASE__SQL_DATABASE__CREDENTIALS__PASSWORD}
POSTGRES_DB: ${EXTRACT_DB_DELTA__SOURCES__SQL_DATABASE__SQL_DATABASE__CREDENTIALS__DATABASE}
ports:
- "${EXTRACT_DB_DELTA__SOURCES__SQL_DATABASE__SQL_DATABASE__CREDENTIALS__PORT}:5432"
volumes:
- ./init.sql:/docker-entrypoint-initdb.d/init.sql

volumes:
postgres_data:
```

init.sql
```sql
CREATE TABLE users (
id SERIAL PRIMARY KEY,
name VARCHAR(100),
email VARCHAR(100),
credit NUMERIC,
list_of_ids INT[]
);

CREATE TABLE products (
id SERIAL PRIMARY KEY,
name VARCHAR(100),
price DECIMAL(10,2)
);

INSERT INTO users (name, email, credit,list_of_ids) VALUES
('John Doe', 'john@test.com',0.999999999, '{1,2}'),
('Jane Smith', 'jane@test.com',0.9999999, '{3}');

INSERT INTO products (name, price) VALUES
('Laptop', 999.99),
('Phone', 599.99),
('Tablet', 299.99);
```

python script
```python
table_config = {"table_name": "users"}

pipeline = dlt.pipeline(
pipeline_name="extract_db_delta",
destination='bigquery',
dataset_name=DATASET_NAME,
progress="log",
)
credentials = ConnectionStringCredentials()

extra_table_args = {}
extra_sql_args = {
# "table_adapter_callback": add_dw_extraction_time_column,
}

sources = sql_database(
credentials=credentials,
chunk_size=10000 * 5,
backend='pyarrow',
include_views=True,
**extra_sql_args,
).with_resources(table_config["table_name"])

logging.info(f"Adding table: {table_config}")

table = sources.resources[table_config["table_name"]]
table.apply_hints(write_disposition="append", **extra_table_args)

info = pipeline.run(sources)
logging.info(info)
```

### Operating system

Linux

### Runtime environment

Local

### Python version

3.11

### dlt data source

sql_database

### dlt destination

Google BigQuery

### Other deployment details

I removed the name of my company from the code so it may have some issues on the environment variables or the variables names.

### Additional information

# Result
![image](https://github.com/user-attachments/assets/71a82f81-b1b7-4dc3-b764-f2836e016da0)

Contributor guide

Open the contributing guide

Research direction

Use the supplied init.sql schema and Python pipeline as the reproduction entry points; run the pyarrow-backed sql_database load into BigQuery and inspect how list_of_ids INT[] is represented. Done means the PostgreSQL ARRAY maps to the intended BigQuery type rather than a nested RECORD, with the behavior verified against the reproduction case.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, postgresql, python, sql
Domain
backend, data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.