dbt-labs / dbt-labs/dbt-adapters
[Feature] make it possible to attibute S3 GET/List requests to specific queries.
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this your first time submitting a feature request?
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
### Describe the feature
When trying to trace down the origins of S3 GET requests it is currently quite difficult to attribute a given GET request with a specific Athena query.
~We already set the User-Agent for all requests (#49) which is awesome and means you can tell that they are coming from the adapter; my proposal is to take that a step further and potentially include some kind of identifier for the exact query being run? Unfortunately this can't be the query execution ID from Athena but perhaps we could inject the ClientRequestToken into the User-Agent header value for the StartQueryExecution call?~
~That way the ClientRequestToken of the original StartQueryExecution call will show up in CloudTrail logs for the S3 GET requests and therefore provide a lineage chain from StartQueryExecution call to GetObject request and back again (check the `responseElements` of the CloudTrail log for the StartQueryExecution request to get the `queryExecutionId`).~
Never mind, see comment. Also it doesn't need to be the ClientRequestToken it can just be anything.
### Describe alternatives you've considered
Haven't really thought of any. Ideas welcome!
### Who will this benefit?
Folks trying to use the adapter at any kind of scale.
### Are you interested in contributing this feature?
Potentially? There's a decent chance I just try to do this on a local fork.
### Anything else?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.