apache / apache/airflow

Allow disabling hook-level lineage in Hook constructors

Open
#63,371 3 comments 0 reactions 0 assignees View on GitHub
area:providers kind:feature needs-triage provider:amazon
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

The S3Hook registers an output asset for every file which it uploads:
https://github.com/apache/airflow/blob/e491aac92a1d50f322a08d92d83616e7c79b3f2e/providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py#L1376-L1379

Which is not always the desired behaviour when using the S3 Hook (see motivating example below).

I'd propose just adding a switch to the S3Hook so disable this sort of lineage:
```python
hook = S3Hook(enable_hook_level_lineage=False)
```

I am happy to submit a fix here, but I wanted to run it by y'all first to make sure that I'm not missing some previous context or undoing an intentional design decision.

### Use case/motivation

We have seen issues where users upload chunked data to S3 within a PythonOperator like so:
```python
hook = S3Hook()
for idx, data in enumerate(list_of_values):
hook.upload_string(data, f"some_prefix/file_{idx}.txt", "some_bucket")
```

Which then creates a _ton_ of output assets. I know that this is limited to 100 output objects (since https://github.com/apache/airflow/pull/45798), but it would be nice if we could disable hook-level lineage altogether and instead manage our own output asset definition at the custom operator / PythonOperator level.

In this case, we likely want to only enable a single output asset at the `some_prefix/` level, not one per file.

This likely applies to other object storage hooks as well.

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.