dlt-hub / dlt-hub/dlt

Oracle Sqlalchemy destination varchar is created as clob

Open
#3,554 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### dlt version

1.20.0

### Describe the problem

When I try to load data from Postgresql to Oracle using Sqlalchemy destination, all varchar fields are converted to CLOB type. When I try same operation with Sling Data, it correctly creates the varchar fields with lengths.

This is my code:
```
def load_psql_to_oracle() -> None:
oracledb.init_oracle_client()

source = sql_database(schema="public",
backend='pyarrow',
chunk_size=100000).with_resources("table1")
pipeline = dlt.pipeline(
pipeline_name="load_psql_to_oracle",
destination="sqlalchemy",
dataset_name="dest_schema",
progress="log"
)
load_info = pipeline.run(source,
write_disposition="replace"
)
print(load_info)
```

The table DDL for Postgrsql:
```sql
create table table1(id int, name varchar(10), surname varchar(20));
```

The table created by DLT on Oracle:
```sql
CREATE TABLE DEST_SCHEMA.TABLE1
(
ID NUMBER(19),
NAME CLOB,
SURNAME CLOB,
"_dlt_load_id" CLOB NOT NULL,
"_dlt_id" VARCHAR2(128 CHAR) NOT NULL
)
```

### Expected behavior

Column `name` should be `VARCHAR2(10)` and `surname` should be `VARCHAR2(20)`

### Steps to reproduce

1. Create a Postgreql table in Postgres: `create table table1(id int, name varchar(10), surname varchar(20));`
2. Insert dummy data: `insert into table1 values(5, "foo", "bar");`
3. Run the provided code above

### Operating system

Linux

### Runtime environment

Local

### Python version

3.10

### dlt data source

Postgresql (sql_database)

### dlt destination

_No response_

### Other deployment details

Destination: Oracle DB using sqlalchemy destination

### Additional information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the provided PostgreSQL-to-Oracle load with the SQLAlchemy destination, then trace the destination's column-type mapping for varchar fields. Confirm the fix by ensuring varchar(10) and varchar(20) produce VARCHAR2(10) and VARCHAR2(20) rather than CLOB.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python, sqlalchemy
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.