apache / apache/hudi

[BUG] hudi-azure package dependencies preventing hudi write operations

Open
#19,729 1 comment 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**
when using hudi-azure 1.2.0 alongside hudi 1.2.0 on spark 4.0 and 4.1 via spark connect, the internal timeline server fails to start upon a write operation, with the error message suggesting a shading issue:
Image
when we --exclude-packages "org.apache.hudi:hudi-timeline-service", which is a dependency of hudi-azure, the timeline starts as it should upon a write operation:
Image

**What you expected:**
hudi-azure 1.2.0 should shade its dependencies such that other packages aren't affected. Or, if I'm misunderstanding the problem: hudi-azure 1.2.0 shouldn't prevent hudi's internal timeline server from starting.

**Steps to reproduce:**
1. write to an azure-based hudi table using spark 4.0.x via spark connect with hudi 1.2.0 and hudi-azure 1.2.0
2. This may require a second write if initialization doesn't kick off the internal timeline server, but as far as I've seen any attempt to start the internal timeline server fails due to this casting failure

### Environment

**Hudi version:** 1.2.0
**Query engine:** Spark 4.0.1
**Relevant configs:**
hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.StorageBasedLockProvider
hoodie.embed.timeline.server=true

### Logs and Stack Trace

shown in "what happened" above

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing an Azure-based Hudi write through Spark Connect with hudi 1.2.0, hudi-azure 1.2.0, and the embedded timeline server enabled. Compare the dependency behavior with org.apache.hudi:hudi-timeline-service excluded, then trace the internal timeline server startup and casting failure. Done means hudi-azure no longer prevents the internal timeline server from starting during writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, java, spark
Domain
backend, data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.