stackabletech / stackabletech/spark-k8s-operator
Iceberg jobs with with Hive 4.0.1 failure
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 72
- Forks
- 4
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 15
Description
Affected Stackable version
all supported
Affected Apache Spark-on-Kubernetes version
No response
Current and expected behavior
Hive 4.0.1 has removed some deprecated apis but the spark iceberg runtime has not been updated.
Spark jobs using the Hive 4.0.1 catalog to write using the iceberg format will fail with the error message:
spark Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table'
Discovered during the upgrade of the anomaly detection demo.
Possible solution
No response
Additional context
No response
Environment
No response
Would you like to work on fixing this bug?
None
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file or test is named in the issue. Start by reproducing a Spark job that writes Iceberg data through the Hive 4.0.1 catalog and trace the Spark Iceberg runtime dependency or configuration involved. Done means the job writes successfully without the Thrift Invalid method name: 'get_table' failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100