aws-samples / aws-samples/dbt-glue
Error using HUDI with dbt-glue: `HoodieException: 'hoodie.table.name' must be set`
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
### Describe the bug
Running `dbt run` on a simple dbt mode with dbt-glue gives `HoodieException: 'hoodie.table.name' must be set` when trying to use Apache HUDI.
### Steps To Reproduce
HUDI installed via JAR and a custom connector in AWS Glue (code is running in GovCloud where AWS Marketplace extensions are not available).
* [HUDI Jar](https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3-bundle_2.12/0.12.1/hudi-spark3-bundle_2.12-0.12.1.jar)
* Class name: org.apache.hudi
* Connection name: hudi_connection
Profiles.yml:
```
govcloud_demo:
outputs:
dev:
type: glue
query-comment: Glue DBT
role_arn: role
region: us-gov-west-1
glue_version: "3.0"
workers: 2
worker_type: G.1X
idle_timeout: 10
schema: "analytics"
database: "analytics"
session_provisioning_timeout_in_seconds: 120
location: "s3://data/path/"
connections: hudi_connection
conf: "spark.serializer=org.apache.spark.serializer.KryoSerializer"
default_arguments: "--enable-metrics=true, --enable-continuous-cloudwatch-log=true, --enable-continuous-log-filter=true, --enable-spark-ui=true, --spark-event-logs-path=s3://logs/path/"
target: dev
```
dbt_project.yml:
```
name: 'govcloud_demo'
version: '1.0.0'
config-version: 2
profile: 'govcloud_demo'
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
target-path: "target"
clean-targets:
- "target"
- "dbt_packages"
models:
+file_format: hudi
govcloud_demo:
example:
+materialized: view
```
Model.sql: (note: Have used file_format=hudi here - same behavior occurs whether configured in dbt_project.yml or in the model file).
```
{{ config(materialized='table') }}
with source_data as (
select 1 as id,
"b" AS anothercol
)
select *
from source_data
```
### Expected behavior
Model runs and creates in Glue catalog / S3.
### Screenshots and log output
```
22:45:01 '''), Py4JJavaError: An error occurred while calling o86.sql.
22:45:01 : org.apache.hudi.exception.HoodieException: 'hoodie.table.name' must be set.
22:45:01 at org.apache.hudi.common.config.HoodieConfig.getStringOrThrow(HoodieConfig.java:237)
22:45:01 at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:98)
22:45:01 at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:144)
22:45:01 at org.apache.spark.sql.execution.datasources.DataSource.writeAndRead(DataSource.scala:530)
...
```
### System information
**The output of `dbt --version`:**
```
Core:
- installed: 1.3.1
- latest: 1.3.1 - Up to date!
Plugins:
- spark: 1.3.0 - Up to date!
```
**The operating system you're using:** MacOS
**The output of `python --version`:** Python 3.10.8
Contributor guide
Research direction
Start by reproducing the failure with the provided profiles.yml, dbt_project.yml, and model.sql using dbt run and the listed Hudi JAR and Glue connection. Trace how dbt-glue passes the Hudi model configuration into AWS Glue and identify where the required table name is lost. Done means the example model runs and creates the expected Glue catalog and S3 output without the HoodieException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100