aws-samples / aws-samples/dbt-glue

conf properties silently ignored in Glue Interactive Sessions: S3FileIO initialized before --conf is applied

Open
#682 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

# [Bug] conf properties silently ignored in Glue Interactive Sessions: S3FileIO initialized before --conf is applied, causing ConnectionPoolTimeoutException on large Iceberg tables

### Describe the bug

When using dbt-glue with Glue Interactive Sessions, all Spark conf properties passed via the `conf` block in `profiles.yml` are silently ignored by S3FileIO, regardless of property name or value. This makes it impossible to configure the S3FileIO HTTP client (e.g. connection pool size or client type) when using the Iceberg catalog, and causes `ConnectionPoolTimeoutException` on any dbt model that reads a large Iceberg table.

Two bugs compound to produce this behaviour:

**Bug 1 — `--conf` string is only partially parsed by Glue Interactive Sessions**

dbt-glue builds `DefaultArguments` for the session with the entire conf block as a single concatenated string under the `--conf` key (`connection.py` lines 165-166):

```python
DefaultArguments = {
"--enable-glue-datacatalog": "true",
"--conf": "spark.sql.extensions=org.apache.iceberg...IcebergSparkSessionExtensions "
"--conf spark.sql.defaultCatalog=glue_catalog "
"--conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog "
"--conf spark.sql.catalog.glue_catalog.s3.http-client.apache.max-connections=1000 "
"...",
...
}
```

The value starts with the first property directly (no leading `--conf`) and embeds subsequent `--conf` prefixes inline as part of a single string value. Glue Interactive Sessions appear to parse only the **first** `key=value` pair from this string and silently discard everything after the first embedded `--conf` token. The result is that `spark.sql.extensions` is applied (the session starts and Iceberg is active) but all `spark.sql.catalog.glue_catalog.*` properties are never set in SparkConf.

**Bug 2 — `--enable-glue-datacatalog=true` causes SparkCatalog to initialize before conf is applied**

`connection.py` line 156 hardcodes `"--enable-glue-datacatalog": "true"` in `DefaultArguments`. This instructs AWS Glue to bootstrap the `glue_catalog` SparkCatalog — including instantiating S3FileIO and its HTTP client — during session startup, before any user-submitted statements run. By the time `_init_session()` executes the first statement, S3FileIO is already constructed with default settings. No subsequent `spark.conf.set()` call can affect the already-constructed FileIO instance.

**Impact:** Any Iceberg table read that issues concurrent `headObject` calls (via `S3InputFile.getEtag()`) against more than 50 Parquet files exhausts the default 50-connection Apache HTTP pool, producing a `ConnectionPoolTimeoutException` that fails the dbt run.

---

### Steps To Reproduce

1. Use dbt-glue with a Glue Interactive Session profile targeting an Iceberg table in the AWS Glue Catalog.

2. Add any `spark.sql.catalog..s3.*` property to the `conf` block in `profiles.yml`, for example:

```yaml
conf: >-
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
--conf spark.sql.defaultCatalog=glue_catalog
--conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
--conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
--conf spark.sql.catalog.glue_catalog.s3.http-client.apache.max-connections=1000
```

3. Run a dbt model that reads a large Iceberg table (hundreds of Parquet files).

4. Observe `ConnectionPoolTimeoutException` — identical to the failure with the default max-connections of 50, proving the property was never applied.

**Diagnostic confirmation:** We set `max-connections` to 1000 in isolation. The failure was byte-for-byte identical to the default of 50. This proves the conf is never read by S3FileIO regardless of value.

---

### Expected behavior

Conf properties set in `profiles.yml` under the `conf` key — in particular `spark.sql.catalog..s3.http-client.apache.max-connections` — should be applied to the S3FileIO instance used by the Iceberg catalog, allowing users to configure the S3 HTTP client pool size.

---

### Screenshots and log output

```
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
at org.apache.iceberg.aws.s3.S3InputFile.getEtag(S3InputFile.java:181)
at org.apache.iceberg.parquet.ReadConf.newReader(ReadConf.java:269)
at org.apache.iceberg.parquet.ReadConf.(ReadConf.java:107)
at org.apache.iceberg.parquet.VectorizedParquetReader.init(VectorizedParquetReader.java:148)
...
Caused by: ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at software.amazon.awssdk.thirdparty.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(...)
at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:261)
```

---

### System information

**The output of `dbt --version`:**
```
Core:
- installed: 1.10.22
- latest: 1.10.22 - Up to date!

Plugins:
- glue: 1.10.19 - Up to date!
```

**The operating system you're using:** Amazon Linux (AWS Glue managed environment)

**The output of `python --version`:** Python 3.11 (AWS Glue managed, Glue version 5.1; also reproduced on Glue 5.0)

---

### Additional context

**Related bug in `_string_to_dict` (default_arguments parsing):**

`connection.py` lines 389-393 parse `default_arguments` by calling `.replace(' ', '')` before splitting on commas. This strips spaces from values, which corrupts any argument value containing spaces. More relevantly: when a user sets `default_arguments: "--enable-glue-datacatalog=false"` to override the hardcoded default, the override is silently dropped unless the value is passed as `",--enable-glue-datacatalog=false"` (with a leading comma to survive the replace+split). This is a separate bug in the same parsing code path.

**Proposed fix:**

Two coordinated changes are needed:

_1. Fix `--conf` string construction_ so each property is passed as a separate `--conf key=value` argument that Glue can parse individually, rather than one concatenated string.

_2. Inject `spark.conf.set()` calls before the USE statement in `_init_session()`_, executed between the SQLPROXY step and the `spark.sql('use ')` step. This sets the catalog conf properties before `SparkCatalog.initialize()` is triggered by the USE statement. `spark.conf.set()` already works inside an Interactive Session (it is used on line 429 of SQLPROXY to set `spark.sql.crossJoin.enabled`), so the mechanism is known-good:

```python
# connection.py -- in _init_session(), after SQLPROXY, before USE statement:
conf_lines = [
line.strip() for line in self.credentials.conf.split("--conf")
if "=" in line.strip() and line.strip().startswith("spark.sql.catalog.")
]
if conf_lines:
set_code = "\n".join(
f'spark.conf.set("{kv.split("=",1)[0].strip()}", "{kv.split("=",1)[1].strip()}")'
for kv in conf_lines
)
GlueStatement(client=self.client, session_id=self.session_id,
code=set_code, poll_interval=...).execute()
# spark.sql('use schema') now triggers SparkCatalog.initialize()
# which reads the freshly-set conf properties
```

With both fixes applied, `SparkCatalog.initialize()` will read `spark.sql.catalog.glue_catalog.s3.http-client.apache.max-connections=1000` and construct S3FileIO with a 1000-connection pool, which is sufficient to handle ~640 concurrent `headObject` calls per executor.

Contributor guide

Open the contributing guide

Research direction

Start in connection.py around lines 156, 165-166, 389-393, and 429, then trace _init_session() through SQLPROXY and the USE statement. Reproduce the Glue Interactive Session behavior with the profiles.yml conf block and check whether each property is parsed and applied before the catalog initializes. Done means catalog S3 settings reach S3FileIO instead of being silently ignored, including the configured connection-pool value.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.