apache / apache/gravitino

[Bug report] Spark DataSource shortcut syntax fails when GravitinoSparkPlugin is enabled

Open
#10,825 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 15h
Merged PRs (30d)
315

Description

### Version

main branch

### Describe what's wrong

When `GravitinoSparkPlugin` is registered in a Spark session, Spark SQL's built-in DataSource shortcut syntax — `SELECT * FROM .\`\`` — fails with a server-side `IllegalArgumentException` instead of falling back to Spark's DataSource resolver.

The shortcut applies to all built-in Spark formats (`parquet`, `csv`, `json`, `orc`, `text`, `avro`, `binaryFile`). Spark parses `.\`\`` as a multipart identifier whose first part is the format name. The Gravitino Spark Connector's `BaseCatalog` intercepts the lookup and forwards it to the Gravitino server. The server-side authorization filter then calls `MetadataObjects.of(TABLE, names)` which requires `names.length == 3` (catalog.schema.table); the multipart identifier has only 2 parts, so the check throws.

Before the plugin is registered, the same query works in vanilla Spark because the analyzer's DataSource shortcut resolver handles the format name directly.

### Error message and/or stacktrace

\`\`\`
Authorization failed due to system internal error. Please contact administrator.
java.lang.IllegalArgumentException: If the type is TABLE, the length of names must be 3
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
at org.apache.gravitino.MetadataObjects.of(MetadataObjects.java:116)
at org.apache.gravitino.MetadataObjects.parse(MetadataObjects.java:184)
at org.apache.gravitino.MetadataObjects.of(MetadataObjects.java:93)
at org.apache.gravitino.utils.NameIdentifierUtil.toMetadataObject(NameIdentifierUtil.java:628)
at org.apache.gravitino.server.authorization.expression.AuthorizationExpressionEvaluator...
...
at org.apache.gravitino.spark.connector.catalog.BaseCatalog.loadGravitinoTable(BaseCatalog.java:437)
at org.apache.gravitino.spark.connector.catalog.BaseCatalog.loadTable(BaseCatalog.java:245)
at org.apache.spark.sql.connector.catalog.CatalogV2Util\$.getTable(CatalogV2Util.scala:363)
at org.apache.spark.sql.catalyst.analysis.Analyzer\$ResolveRelations\$...
\`\`\`

### How to reproduce

1. Gravitino server (main branch), a metalake with authorization enabled.
2. Spark 3.5 session with:
\`\`\`
spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin
spark.sql.gravitino.uri=
spark.sql.gravitino.metalake=
\`\`\`
3. Any file accessible by Spark (local, s3a, gvfs, hdfs, etc.).
4. Run:
\`\`\`sql
SELECT * FROM parquet.\`s3a://bucket/file.parquet\`;
-- or csv, json, orc, text, avro — same failure
\`\`\`

Expected: Spark reads the file via \`HadoopFsRelation\`.

Actual: \`IllegalArgumentException\` "If the type is TABLE, the length of names must be 3".

### Additional context

The shortcut syntax is a long-standing public Spark API (since 2.0). Workarounds today are: (a) \`CREATE TEMPORARY VIEW ... USING OPTIONS (path=...)\`, or (b) use the DataFrame API \`spark.read.format(...).load(path)\`. Both bypass the Gravitino catalog registration and succeed.

Discovered against Gravitino 1.2.0 in a Spark Connect + Kyuubi shared engine setup using GVFS paths, but the bug is format-agnostic and path-agnostic — any built-in format + any path reproduces it.

Contributor guide

Open the contributing guide

Research direction

Reproduce the Spark 3.5 query with GravitinoSparkPlugin enabled, then trace BaseCatalog.loadTable and loadGravitinoTable in the connector alongside MetadataObjects.of/parse and NameIdentifierUtil.toMetadataObject in the server. Determine how the two-part DataSource shortcut is intercepted and authorized. Done means built-in format shortcuts resolve through Spark without the IllegalArgumentException, while normal catalog table lookups remain unaffected.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
authorization, backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.