aws-samples / aws-samples/dbt-glue

Session management fails with tag-based IAM permissions in SageMaker Unified Studio

Open
#556 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

## Summary
When using tag-based IAM permissions in AWS SageMaker Unified Studio environments, the `_recreate_session` method fails with `AccessDeniedException` when attempting to delete sessions. This is particularly problematic because `_recreate_session` is called during the initial connection establishment, causing dbt to fail completely in SageMaker Unified Studio environments. Since tag-based permissions can cause this exception even when the session doesn't exist, the method should attempt session creation anyway and let the create operation handle cases where the session actually exists.

## Environment
- **AWS Environment**: SageMaker Unified Studio
- **IAM Permission Model**: Tag-based (sessions must have `AmazonDataZoneProject` tag matching principal tag)

## Problem Description

### Current Behavior
The `_recreate_session` method always attempts to delete an existing session before creating a new one:

```python
def _recreate_session(self, session_id: str) -> None:
logger.debug("GlueConnection _recreate_session called")
self.delete_session(session_id=session_id) # This fails with AccessDeniedException
logger.debug(f'Deleted session with id {session_id}')
self._create_session(session_id=session_id)
```

### Issue
In SageMaker Unified Studio's tag-based IAM environments, users can only delete sessions that have the required tags (e.g., `AmazonDataZoneProject: xyz`). When trying to delete a non-existent session or a session without proper tags, AWS returns `AccessDeniedException`.

### Error Example
```
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the DeleteSession operation: User is not authorized to perform: glue:DeleteSession
```

## Expected Behavior
The session recreation should attempt to create a new session when `AccessDeniedException` occurs during deletion, since this likely indicates the session doesn't exist. If the session actually does exist, the subsequent `create_session` call will properly fail with an appropriate exception (e.g., `ResourceAlreadyExistsException`), which is the expected behavior.

## Root Cause
SageMaker Unified Studio uses tag-based IAM policies to control resource access. These policies restrict session operations based on tag matching:

```json
{
"Condition": {
"StringEquals": {
"aws:ResourceTag/AmazonDataZoneProject": "${aws:PrincipalTag/AmazonDataZoneProject}"
}
}
}
```

This means:
- If session doesn't exist → `AccessDeniedException`
- If session exists but lacks required tags → `AccessDeniedException`
- Only if session exists AND has matching tags → deletion succeeds

## Proposed Solution
Modify `_recreate_session` to handle `AccessDeniedException` gracefully, treating it as an indication that the session likely doesn't exist and we should attempt creation:

```python
def _recreate_session(self, session_id: str) -> None:
"""Deletes any existing session with session_id and creates a new one."""
logger.debug("GlueConnection _recreate_session called")

try:
self.delete_session(session_id=session_id)
logger.debug(f'Deleted session with id {session_id}')
except self.client.exceptions.AccessDeniedException:
# In tag-based permission environments, AccessDeniedException could mean
# the session doesn't exist, so we should attempt creation
logger.debug(f"AccessDeniedException when trying to delete session {session_id}. "
f"Session likely doesn't exist. Attempting creation.")

self._create_session(session_id=session_id)
```

This approach allows the normal error handling flow to work: if the session truly exists, `create_session` will throw `ResourceAlreadyExistsException` or similar.

## Workaround
Currently, users must ensure their IAM policies allow deletion of any session, which may not be acceptable in security-conscious environments.

I'm relatively new to dbt-glue and would appreciate feedback on this proposed solution. Are there any potential issues or edge cases I might be missing with this approach?

Contributor guide

Open the contributing guide

Research direction

Start by locating the GlueConnection._recreate_session entry point and reading how delete_session and _create_session handle client exceptions. Confirm that an AccessDeniedException during deletion still reaches creation, while existing-session errors from creation remain visible; verify the relevant connection behavior with the project's available checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.