aws-samples / aws-samples/dbt-glue
dbt Glue should be invokable from Cross Acount workers
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
### Describe the feature
As issue #634 illustrates, cross-account dbt-glue invocation is a common and
currently broken use case. When a worker (e.g. Airflow, Dagster) runs in one AWS
account but Glue resources live in another, the adapter must assume role_arn via STS
before making any Glue API calls.
This is especially painful for workers running on Kubernetes with IRSA or EKS Pod
Identity, where assuming a cross-account role externally — before invoking dbt — is
not viable. The worker's IAM role has permission to assume the destination account's
role, but there is no clean hook to do that assumption prior to dbt taking over.
The adapter must handle it internally.
The problem is worse for orchestrators like Dagster, which provide convenience
functions (e.g. @dbt_assets) to register all dbt models as software-defined assets.
These abstractions offer less flexibility than a raw Airflow PythonOperator or Bash
command — there is no obvious place to inject a cross-account assume-role step
before the adapter initializes. The adapter itself must be configurable to perform
the role assumption.
A credential flag use_interactive_session_role_for_api_calls already exists in the
adapter for this purpose, and is partially wired: it correctly assumes role_arn
before making Glue management API calls (DDL, schema operations) in
impl.py::get_connection(). However, GlueConnection.client — the boto3 client used
for interactive session lifecycle (CreateSession, GetSession, StopSession,
DeleteSession) — never applies the same logic. This means cross-account users hit an
AccessDeniedException at session creation even when the flag is set.
The fix is to apply the same sts:AssumeRole logic in GlueConnection.client,
completing the half-implemented flag.
### Describe alternatives you've considered
Externally assuming the role before invoking dbt — not viable for IRSA/EKS Pod
Identity workers. The assumed credentials would need to be injected into the
environment before the dbt process starts, which requires wrapping every invocation
in a credential-refresh shim. This is fragile and breaks for orchestrators like
Dagster that abstract away the invocation entirely.
A new flag (e.g. assume_cross_account_role) — considered, but unnecessary. The
existing use_interactive_session_role_for_api_calls flag already captures the
correct intent ("assume the role before making Glue API calls"). Adding a second
flag would require users to set both flags for full cross-account support, which is
confusing. Completing the existing flag is the right approach.
### Additional context
The AWS Glue Interactive Sessions security model uses two principals (docs):
- Client principal — the IAM identity running dbt. Makes Glue API calls
(CreateSession, RunStatement, etc.). Requires glue:* + iam:PassRole to the runtime
role.
- Runtime role (role_arn) — passed to CreateSession. AWS Glue itself assumes this to
execute Spark. Requires S3, CloudWatch, etc.
In cross-account scenarios (client principal in account A, Glue resources in account
B), the client principal must first assume role_arn via STS before it can call any
Glue API. Setting use_interactive_session_role_for_api_calls: true in profiles.yml
should make the adapter handle this transparently.
Issue #634 shows the exact failure: an Airflow worker with EKS Pod Identity calls
GetDatabases as its source-account role rather than the assumed destination-account
role, resulting in AccessDeniedException. This is reproducible with any
cross-account role_arn in profiles.yml without the flag — and with the flag set, it
still fails at session creation because GlueConnection.client is unwired.
### Who will this benefit?
Any team running dbt-glue from a worker in a different AWS account than their Glue
resources. Specifically:
- Multi-account AWS organizations where compute (Airflow, Dagster, Prefect, GitHub
Actions) lives in a tooling/CICD account and data resources live in separate data or
environment accounts.
- Kubernetes-based workers (EKS with IRSA or Pod Identity) where injecting assumed
credentials before dbt starts is not possible without custom wrapper code.
- Dagster users who use @dbt_assets to register dbt models as software-defined
assets — Dagster invokes dbt internally, leaving no hook to perform a cross-account
assume before the adapter initializes.
### Are you interested in contributing this feature?
Yes — a PR is already open: #676. It completes the
use_interactive_session_role_for_api_calls flag for the session management client,
adds a unit test, and documents the flag in the README.
Contributor guide
Research direction
Work is already underway in PR #676. Start with GlueConnection.client and the existing use_interactive_session_role_for_api_calls handling in impl.py::get_connection(); review the added unit test and README changes. Done means the session lifecycle calls use the assumed role and the flag is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100