[BUG] hudi-azure package dependencies preventing hudi write operations
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:**
when using hudi-azure 1.2.0 alongside hudi 1.2.0 on spark 4.0 and 4.1 via spark connect, the internal timeline server fails to start upon a write operation, with the error message suggesting a shading issue:
when we --exclude-packages "org.apache.hudi:hudi-timeline-service", which is a dependency of hudi-azure, the timeline starts as it should upon a write operation:
**What you expected:**
hudi-azure 1.2.0 should shade its dependencies such that other packages aren't affected. Or, if I'm misunderstanding the problem: hudi-azure 1.2.0 shouldn't prevent hudi's internal timeline server from starting.
**Steps to reproduce:**
1. write to an azure-based hudi table using spark 4.0.x via spark connect with hudi 1.2.0 and hudi-azure 1.2.0
2. This may require a second write if initialization doesn't kick off the internal timeline server, but as far as I've seen any attempt to start the internal timeline server fails due to this casting failure
### Environment
**Hudi version:** 1.2.0
**Query engine:** Spark 4.0.1
**Relevant configs:**
hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.StorageBasedLockProvider
hoodie.embed.timeline.server=true
### Logs and Stack Trace
shown in "what happened" above
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing an Azure-based Hudi write through Spark Connect with hudi 1.2.0, hudi-azure 1.2.0, and the embedded timeline server enabled. Compare the dependency behavior with org.apache.hudi:hudi-timeline-service excluded, then trace the internal timeline server startup and casting failure. Done means hudi-azure no longer prevents the internal timeline server from starting during writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, java, spark
- Domain
- backend, data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100