aws-samples / aws-samples/dbt-glue

Unable to connect to glue catalog in different AWS project from current AWS project

Open
#174 5 comments 0 reactions 0 assignees View on GitHub
bug enhancement
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the bug

I am trying to run dbt model from current AWS project and I am getting `AnalysisException: Table or view not found: `xxx.yyyy` as my source is in different AWS project glue catalog. Though i have permissions to read other AWS project catalog, still I am unable to read table from different catalog.

### Steps To Reproduce

My source looks like as below

```
version: 2

sources:
- name: datalake
schema: secret
tables:
# please maintain alphabetical order
- name: secret1
```

My model looks like following

```
{{
config(
materialized="incremental",
file_format='parquet',
incremental_strategy='insert_overwrite',
table_properties={'write.target-file-size-bytes': '268435456'}
)
}}

with base as
(
select * from {{ source('datalake', 'secret1') }}
where a=123
)

SELECT *
FROM base
```

My target connection looks like as follows where iam explicitly passing catalogid to spark conf

```
dev:
type: glue
role_arn: arn:aws:iam::xxxxxx:role/ccoeGlueEDADBTExecution
region: us-east-2
workers: 10
glue_version: "3.0"
worker_type: G.2X
idle_timeout: 10
schema: "{{ env_var('SCHEMA', 'dbt_wsb') }}"
conf: --conf hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
--conf hive.metastore.glue.catalogid=xxxx --conf aws.region=us-east-2
session_provisioning_timeout_in_seconds: 120
location: "{{ env_var('LOCATION', 's3://dbt-wsb/models') }}"

```

When I run dbt model, I get following error

```
AnalysisException: Table or view not found: `secret.secret1`; line 20 pos 18;
'CreateTable `smullangi_utils`.`datalake_model`, ErrorIfExists
+- 'Project [*]
+- 'SubqueryAlias base
+- 'Project [*]
+- 'Filter ('advertisement_id = 123)
+- 'UnresolvedRelation [secret.secret1], [], false
```

### Expected behavior
Model should be created as we passed catalogid explicity. We need to know exact way to pass catalogid to source so that my dbt understands and creates table.

### Screenshots and log output
If applicable, add screenshots or log output to help explain your problem.

### System information
**The output of `dbt --version`:**
```
Core:
- installed: 1.3.4
- latest: 1.5.0 - Update available!

Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation

Plugins:
- glue: 1.3.12 - Update available!
- spark: 1.3.1 - Update available!

At least one plugin is out of date or incompatible with dbt-core.
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
```

**The operating system you're using:**
macos 13.3.1 (22E261)

**The output of `python --version`:**
Python 3.9.7

### Additional context
We have run glue job from AWS console using python and is working from our project. We are able to read catalog from different AWS project from current AWS project. This tells us that there is no problem with provided access/policies. The problem is only when we run with dbt-glue adapter locally.
I followed all approaches mentioned here https://repost.aws/knowledge-center/query-glue-data-catalog-cross-account but still its not working (passed all parameters in conf of profiles.yml)
Also we are able to run dbt model using dbt-glue adapter when source table is in current catalog.

Contributor guide

Open the contributing guide

Research direction

Start with the dbt-glue adapter's local Glue connection and the profiles.yml conf shown in the issue. Reproduce the model against a cross-account AWS Glue source, then compare it with the working same-catalog case. Done means the source table resolves and the model is created using the specified catalog ID.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, spark
Domain
cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.