[Bug report] Spark DataSource shortcut syntax fails when GravitinoSparkPlugin is enabled
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 315
Description
### Version
main branch
### Describe what's wrong
When `GravitinoSparkPlugin` is registered in a Spark session, Spark SQL's built-in DataSource shortcut syntax — `SELECT * FROM .\`\`` — fails with a server-side `IllegalArgumentException` instead of falling back to Spark's DataSource resolver.
The shortcut applies to all built-in Spark formats (`parquet`, `csv`, `json`, `orc`, `text`, `avro`, `binaryFile`). Spark parses `.\`\`` as a multipart identifier whose first part is the format name. The Gravitino Spark Connector's `BaseCatalog` intercepts the lookup and forwards it to the Gravitino server. The server-side authorization filter then calls `MetadataObjects.of(TABLE, names)` which requires `names.length == 3` (catalog.schema.table); the multipart identifier has only 2 parts, so the check throws.
Before the plugin is registered, the same query works in vanilla Spark because the analyzer's DataSource shortcut resolver handles the format name directly.
### Error message and/or stacktrace
\`\`\`
Authorization failed due to system internal error. Please contact administrator.
java.lang.IllegalArgumentException: If the type is TABLE, the length of names must be 3
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
at org.apache.gravitino.MetadataObjects.of(MetadataObjects.java:116)
at org.apache.gravitino.MetadataObjects.parse(MetadataObjects.java:184)
at org.apache.gravitino.MetadataObjects.of(MetadataObjects.java:93)
at org.apache.gravitino.utils.NameIdentifierUtil.toMetadataObject(NameIdentifierUtil.java:628)
at org.apache.gravitino.server.authorization.expression.AuthorizationExpressionEvaluator...
...
at org.apache.gravitino.spark.connector.catalog.BaseCatalog.loadGravitinoTable(BaseCatalog.java:437)
at org.apache.gravitino.spark.connector.catalog.BaseCatalog.loadTable(BaseCatalog.java:245)
at org.apache.spark.sql.connector.catalog.CatalogV2Util\$.getTable(CatalogV2Util.scala:363)
at org.apache.spark.sql.catalyst.analysis.Analyzer\$ResolveRelations\$...
\`\`\`
### How to reproduce
1. Gravitino server (main branch), a metalake with authorization enabled.
2. Spark 3.5 session with:
\`\`\`
spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin
spark.sql.gravitino.uri=
spark.sql.gravitino.metalake=
\`\`\`
3. Any file accessible by Spark (local, s3a, gvfs, hdfs, etc.).
4. Run:
\`\`\`sql
SELECT * FROM parquet.\`s3a://bucket/file.parquet\`;
-- or csv, json, orc, text, avro — same failure
\`\`\`
Expected: Spark reads the file via \`HadoopFsRelation\`.
Actual: \`IllegalArgumentException\` "If the type is TABLE, the length of names must be 3".
### Additional context
The shortcut syntax is a long-standing public Spark API (since 2.0). Workarounds today are: (a) \`CREATE TEMPORARY VIEW ... USING OPTIONS (path=...)\`, or (b) use the DataFrame API \`spark.read.format(...).load(path)\`. Both bypass the Gravitino catalog registration and succeed.
Discovered against Gravitino 1.2.0 in a Spark Connect + Kyuubi shared engine setup using GVFS paths, but the bug is format-agnostic and path-agnostic — any built-in format + any path reproduces it.
Contributor guide
Research direction
Reproduce the Spark 3.5 query with GravitinoSparkPlugin enabled, then trace BaseCatalog.loadTable and loadGravitinoTable in the connector alongside MetadataObjects.of/parse and NameIdentifierUtil.toMetadataObject in the server. Determine how the two-part DataSource shortcut is intercepted and authorized. Done means built-in format shortcuts resolve through Spark without the IllegalArgumentException, while normal catalog table lookups remain unaffected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark, sql
- Domain
- authorization, backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100