aws-samples / aws-samples/dbt-glue

Hanging session after python model is complete

Open
#664 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the bug
After successful or failed build of a python dbt model the session still hangs despite not having `glue_session_reuse: true`
mentioned in: #663
**NOTE:** Running from this from my fork in master branch: https://github.com/akmalsoliev/dbt-glue/tree/master

### Steps To Reproduce
Use an example provided and set profile that doesn't have `glue_session_reuse: true`
```py
def model(dbt, spark):
# Configure the model
dbt.config(materialized='python_model', file_format='iceberg')

# Create your DataFrame using Spark
data = [
(1, 'Alice', 100),
(2, 'Bob', 200),
(3, 'Charlie', 300)
]
columns = ['id', 'name', 'value']

# Return a Spark DataFrame
return spark.createDataFrame(data, columns)
```

### Expected behavior
The session is terminated upon completion

### Screenshots and log output

```
10:24:59 Running with dbt=1.10.19
10:25:00 Registered adapter: glue=1.10.15
10:25:00 Unable to do partial parsing because config vars, config profile, or config target have changed
10:25:00 Unable to do partial parsing because profile has changed
10:25:01 Found x models, x data tests, x sources, xx macros
10:25:01
10:25:01 Concurrency: 10 threads (target='prod')
10:25:01
10:25:04 1 of 1 START python incremental model xxx.python_model [RUN]
DEBUG: Using Glue session: dbt-glue-xxxx-560f
10:29:43 1 of 1 OK created python incremental model xxx.python_model [OK in 279.08s]
10:29:43
10:29:43 Finished running 1 incremental model in 0 hours 4 minutes and 41.41 seconds (281.41s).
10:29:43
10:29:43 Completed successfully
10:29:43
10:29:43 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=1
```

Despite passing the test the session is kept alive:
Image

### System information
**The output of `dbt --version`:**
```
Core:
- installed: 1.10.19
- latest: 1.11.5 - Update available!

Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation

Plugins:
- glue: 1.10.15 - Up to date!
- spark: 1.9.3 - Update available!

At least one plugin is out of date with dbt-core.
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
```

**The operating system you're using:**
macOS 26.3 (25D125)

**The output of `python --version`:**
Python 3.12.12

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the provided Python model and a profile that omits glue_session_reuse: true, using the example and the reported dbt and Python versions as a reference. Trace the adapter's Glue session handling after both successful and failed model runs; done means the session terminates when reuse is not enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, spark
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.