Oracle Sqlalchemy destination varchar is created as clob
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### dlt version
1.20.0
### Describe the problem
When I try to load data from Postgresql to Oracle using Sqlalchemy destination, all varchar fields are converted to CLOB type. When I try same operation with Sling Data, it correctly creates the varchar fields with lengths.
This is my code:
```
def load_psql_to_oracle() -> None:
oracledb.init_oracle_client()
source = sql_database(schema="public",
backend='pyarrow',
chunk_size=100000).with_resources("table1")
pipeline = dlt.pipeline(
pipeline_name="load_psql_to_oracle",
destination="sqlalchemy",
dataset_name="dest_schema",
progress="log"
)
load_info = pipeline.run(source,
write_disposition="replace"
)
print(load_info)
```
The table DDL for Postgrsql:
```sql
create table table1(id int, name varchar(10), surname varchar(20));
```
The table created by DLT on Oracle:
```sql
CREATE TABLE DEST_SCHEMA.TABLE1
(
ID NUMBER(19),
NAME CLOB,
SURNAME CLOB,
"_dlt_load_id" CLOB NOT NULL,
"_dlt_id" VARCHAR2(128 CHAR) NOT NULL
)
```
### Expected behavior
Column `name` should be `VARCHAR2(10)` and `surname` should be `VARCHAR2(20)`
### Steps to reproduce
1. Create a Postgreql table in Postgres: `create table table1(id int, name varchar(10), surname varchar(20));`
2. Insert dummy data: `insert into table1 values(5, "foo", "bar");`
3. Run the provided code above
### Operating system
Linux
### Runtime environment
Local
### Python version
3.10
### dlt data source
Postgresql (sql_database)
### dlt destination
_No response_
### Other deployment details
Destination: Oracle DB using sqlalchemy destination
### Additional information
_No response_
Contributor guide
Research direction
Start by reproducing the provided PostgreSQL-to-Oracle load with the SQLAlchemy destination, then trace the destination's column-type mapping for varchar fields. Confirm the fix by ensuring varchar(10) and varchar(20) produce VARCHAR2(10) and VARCHAR2(20) rather than CLOB.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python, sqlalchemy
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100