[Bug report] federation query using 2 hive metastores does not work when using gravitino
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Version
main branch
### Describe what's wrong
I want to use federation query using hive metastore stored in 2 hadoop clusters.
So we added two hive catalogues to metalake.
There is a difference between the location path in the show create table and the actual location information when sql-sql querying.
It seems to be an effect of the actual spark-sql query `spark.sql.metastore.uris` option, so I'm wondering if it's possible to federate query 2 hives?
### Error message and/or stacktrace
```sql
> show create table portal_test_schema;
CREATE TABLE portal_test_schema (
...
month INT,
day INT,
hour INT
)
PARTITIONED BY (month, day, hour)
LOCATION 'hdfs://test1/test1'
TBLPROPERTIES (
'bucketing_version' = '2',
'discover.partitions' = 'true',
'input-format' = 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat',
```
explain query
```sql
spark-sql ()> EXPLAIN show create table portal_test_schema;
== Physical Plan ==
ShowCreateTable [createtab_stmt#0], HiveTable(org.apache.spark.sql.SparkSession@14144cc9,CatalogTable(
Database: ladp
Table: portal_test_schema
Created Time: Thu Jan 26 18:40:15 JST 2023
Last Access: UNKNOWN
Created By: Spark 2.2 or prior
Type: EXTERNAL
Provider: hive
Table Properties: [bucketing_version=2, numFilesErasureCoded=0, transient_lastDdlTime=1725947686]
Location: hdfs://test2/portal_test_schema
Serde Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat: org.apache.hadoop.mapred.TextInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Storage Properties: [serialization.format=1]
Partition Provider: Catalog
Schema: root
...
),org.apache.kyuubi.spark.connector.hive.HiveTableCatalog@64cbc28e)
```
### How to reproduce
gravitino branch main
### Additional context
_No response_
Contributor guide
Research direction
Start by reproducing the setup on the Gravitino main branch with two Hive catalogs backed by separate Hadoop clusters. Compare SHOW CREATE TABLE output and the physical plan from EXPLAIN, then inspect how Spark SQL's spark.sql.metastore.uris setting affects the selected table location. Done means determining whether two Hive metastores can be federated and identifying the location mismatch's cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, spark
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100