apache / apache/airflow

pandas dataframe.to_sql on uuid datatype columns

Open
#71,977 4 comments 0 reactions 0 assignees View on GitHub
kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Task SDK

### Apache Airflow version

3.3.1

### What happened and how to reproduce it?

Using Airflow docker image for data driven solutions.Immediately I upgraded from 3.3.0 to 3.3.1, I got serious bugs. For instance, using postgresql with sqlalchemy create_engine, I realised while inserting data from a dataframe into a postgresql table with a column with data type uuid, the columns of the dataset coming from the "select" statement of the dataframe are all being handles as strings/text, but in postgresql, you can only explicitly insert a uuid element in a uuid column!So trying to insert the string while casting it on the fly to uuid is not working in Airflow 3.3.1. Basically in all the dags in which I made use of sqlalchemy create_engine in defining the postgresql connection hook failed immediately the code got to the point where I made use of dataframe.to_sql. We would surely have to downgrade back to Airflow 3.3.0 until this bug is solved. Thank you.

### What you think should happen instead?

I think there is a major issue with the pandas's dataframe.to_sql method in Airflow3.3.1 and the datatypes of table columns should remain thesame as the source dataframe. Using string casts on the fly is not going to help while working with special data types.

### Operating System

linus ubuntu noble

### Deployment

None

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

3.3.1

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

Not applicable

### Anything else?

_No response_

### Are you willing to submit PR?

- [x] #72001

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

The report names no repository files or tests. Start by reproducing dataframe.to_sql against PostgreSQL UUID columns under Airflow 3.3.0 and 3.3.1, then trace the SQLAlchemy connection path; done means UUID values insert successfully without requiring an incompatible string cast.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, postgresql, python, sqlalchemy
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.