aws-samples / aws-samples/dbt-glue

Dbt runs fail after ~350s using Glue 5.0 with llegalSessionStateException

Open
#576 1 comment 2 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the Bug

All my dbt model runs crash around **5 minutes and 30 seconds** into execution with the following error:

```
Error in GlueCursor execute: An error occurred (IllegalSessionStateException) when calling the GetStatement operation: Session unavailable, fail to call ReplServer.
```

The run writes files to an S3 bucket, however, the table does **not get registered** in the Glue Catalog.

The source tables are registered in the Glue Catalog and configured via `sources.yml`. Example for the `conta` table:

```yaml
- name: conta
description: "Raw CDC data from DMS for conta"
meta:
external_location: s3://zap-ingestion/zapweb/conta/
external:
location: "s3://zap-ingestion/zapweb/conta/"
file_format: "parquet"
```

These tables are built on top of **compacted Parquet files** generated by AWS DMS and processed using https://github.com/awslabs/aws-glue-blueprint-libs/tree/master/samples/compaction . The goal is to build a CDC process and the 1sr run of each table may process from 10MB to 100GB of data.

---

### Steps to Reproduce

#### dbt Model (`brz_conta.sql`)

```sql
{{ config(
tags=["brz"],
materialized='incremental',
incremental_strategy='append',
pre_hook="SET spark.sql.parquet.datetimeRebaseModeInRead = LEGACY"
) }}

WITH source_data AS (
SELECT
CAST(dms_commit_ts AS TIMESTAMP) AS loaded_at,
CAST(idconsumo AS STRING) AS idconsumo,
CAST(idsubscricao AS STRING) AS idsubscricao,
CAST(idprodutoinstantaneo AS STRING) AS idprodutoinstantaneo,
CAST(datainicio AS STRING) AS datainicio,
CAST(datafim AS STRING) AS datafim,
CAST(valor AS STRING) AS valor,
CAST(estado AS STRING) AS estado,
CAST(nagra AS STRING) AS nagra,
CAST(incarencia AS STRING) AS incarencia,
CAST(ideventoconsumo AS STRING) AS ideventoconsumo,
CAST(pontos AS STRING) AS pontos,
CAST(quota AS STRING) AS quota,
CAST(idstb AS STRING) AS idstb,
CAST(idequipamento AS STRING) AS idequipamento,
CAST(idcontaservico AS STRING) AS idcontaservico,
CAST(datahorainicio AS STRING) AS datahorainicio,
CAST(datahorafim AS STRING) AS datahorafim,
CAST(datacriacao AS STRING) AS datacriacao
FROM {{ source('zap_ingestion', 'consumo') }}
{% if is_incremental() %}
WHERE loaded_at > (SELECT MAX(loaded_at) FROM {{ this }})
{% endif %}
)

SELECT * FROM source_data
```

#### dbt `profiles.yml`

```yaml
prod:
type: glue
query_tag: zap-deltalake
role_arn: ROLE
region: af-south-1
workers: 2
worker_type: G.1X
session_provisioning_timeout_in_seconds: 300
idle_timeout: 15
create_new_session: true
default_arguments: "--enable-auto-scaling=true, --enable-metrics=true, --enable-continuous-cloudwatch-log=true, --enable-continuous-log-filter=true, --enable-spark-ui=true, --spark-event-logs-path=s3://BUCKET/prodLogDir/"
glue_version: "5.0"
schema: ice_lake
catalog_id: ID
location: s3://BRZ_BUCKET
datalake_formats: iceberg
custom_iceberg_catalog_namespace: ""
spark_conf:
spark.sql.extensions: org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.glue_catalog: org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.glue_catalog.warehouse: s3://BRZ_BUCKET/ice_lake/
spark.sql.warehouse: s3://BRZ_BUCKET/ice_lake/
spark.sql.catalog.glue_catalog.catalog-impl: org.apache.iceberg.aws.glue.GlueCatalog
spark.sql.catalog.glue_catalog.io-impl: org.apache.iceberg.aws.s3.S3FileIO
spark.serializer: org.apache.spark.serializer.KryoSerializer
```

---

### Expected Behavior

The model `brz_conta` should:
- Create an Iceberg table named `brz_conta` in the `ice_lake` schema.
- Register it in the Glue Catalog.
- Store data files in the specified S3 path.

---

### Screenshots and Log Output

#### DBT Output

```
Glue adapter: Error in GlueCursor (session_id=dbt-glue__service-role/AWSGlueServiceRole__e0ea176d-fa40-4e95-9709-d4ca72d5508a) execute:
An error occurred (IllegalSessionStateException) when calling the GetStatement operation:
Session dbt-glue__service-role/... unavailable, fail to call ReplServer

Runtime Error in model brz_conta (models/bronze/brz_conta.sql):
module 'dbt.exceptions' has no attribute 'ExecutableError'
```

#### AWS Glue Interactive Sessions Logs

There are several successful downloads before the process fails:

```
25/07/10 13:17:56 ERROR AsyncFileDownloader: TID: 246 - Download failed for ParquetFileChunk(path=s3://zap-ingestion/zapweb/conta/20250527-194357672.parquet, downloadSize=30)
java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.fs.FileSystem.hasPathCapability(org.apache.hadoop.fs.Path, String)" because "this.fileSystem" is null
at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.hasPathCapability(EmrFileSystem.java:377) ~[emrfs-hadoop-assembly-2.69.0.jar:?]

ERROR 2025-07-10T13:17:56,872 311398 com.amazonaws.glue.is.LivyServerLauncher [main] 86 Got interrupted
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.InterruptedException: Session did not reach healthy state
```

#### Spark Output Sample

```json
{
"Event": "SparkListenerJobEnd",
"Job ID": 3,
"Completion Time": 1752153476899,
"Job Result": {
"Result": "JobFailed",
"Exception": {
"Message": "Job 3 cancelled because SparkContext was shut down"
}
},
"Metrics Summary": {
"Rows Written": 6143958,
"Bytes Written": 259720476,
"Input Records Read": 6143958,
"Bytes Read": 617032200
}
}
```

---

### System Information

**The output of `dbt --version`:**

```
dbt-core==1.9.6
dbt-glue==1.9.4
```

**The operating system you're using:**

```
Docker image: ghcr.io/dbt-labs/dbt-spark:latest
```

Contributor guide

Open the contributing guide

Research direction

Start with the supplied brz_conta.sql model and profiles.yml using dbt-core 1.9.6, dbt-glue 1.9.4, and Glue 5.0, then compare the dbt output with the Interactive Sessions and Spark logs. Trace the session failure around the 350-second mark and the ExecutableError message. Done means the model completes, creates the Iceberg table in ice_lake, registers it in Glue Catalog, and writes the expected S3 files.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python, sql
Domain
cloud, data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.