dlt-hub / dlt-hub/dlt

Enable dlt to run on managed AWS airflow MWAA

Open
#995 3 comments 0 reactions 0 assignees View on GitHub
bug community
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### Feature description

User:
"I've been at a stand still actually. It looks to me like the constraints from MWAA are going to prohibit me from using Snowflake as a destination. Writing to S3 is a viable replacement- but still having trouble here as well. The most recent version of airflow available on MWAA is 2.7.2.
At the moment, airflow is showing some import conflicts. It can't see the path to locate PipelineTasksGroup in my dag, nor can it find DltResource in init.py.
There is also a version conflict with s3fs which relies on aiobotocore. The constraint on aiobotocore here is 2.6.0 which is not"

### Are you a dlt user?

I'd consider using dlt, but it's lacking a feature I need.

### Use case

Run airflow on managed AWS with dlt. Library conflicts, seems S3 could be easier .

You can ask more information from the user here https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1707496343595039

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the MWAA Airflow 2.7.2 constraints and the reported import paths for PipelineTasksGroup and DltResource in init.py, then examine the s3fs and aiobotocore version conflict. Done means dlt can run on managed AWS MWAA with the stated imports and a viable S3 destination, but the issue does not name repository files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.