Enable dlt to run on managed AWS airflow MWAA
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### Feature description
User:
"I've been at a stand still actually. It looks to me like the constraints from MWAA are going to prohibit me from using Snowflake as a destination. Writing to S3 is a viable replacement- but still having trouble here as well. The most recent version of airflow available on MWAA is 2.7.2.
At the moment, airflow is showing some import conflicts. It can't see the path to locate PipelineTasksGroup in my dag, nor can it find DltResource in init.py.
There is also a version conflict with s3fs which relies on aiobotocore. The constraint on aiobotocore here is 2.6.0 which is not"
### Are you a dlt user?
I'd consider using dlt, but it's lacking a feature I need.
### Use case
Run airflow on managed AWS with dlt. Library conflicts, seems S3 could be easier .
You can ask more information from the user here https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1707496343595039
Contributor guide
Research direction
Start by reviewing the MWAA Airflow 2.7.2 constraints and the reported import paths for PipelineTasksGroup and DltResource in init.py, then examine the s3fs and aiobotocore version conflict. Done means dlt can run on managed AWS MWAA with the stated imports and a viable S3 destination, but the issue does not name repository files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100