[SUPPORT] Hudi CLI bundle not working
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
When using `hudi-cli-with-bundle.sh` to start Hudi CLI, many commands would fail immediately due to error below:
```
24/01/26 00:22:51 INFO InputStreamConsumer: Error: Failed to load org.apache.hudi.cli.commands.SparkMain: org/apache/hudi/common/engine/HoodieEngineContext
24/01/26 00:22:51 INFO InputStreamConsumer: 24/01/26 00:22:51 INFO ShutdownHookManager: Shutdown hook called
24/01/26 00:22:51 INFO InputStreamConsumer: 24/01/26 00:22:51 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-5465e2a8-7b9a-4cf1-b46a-afec9f50d860
Failed to clean hoodie dataset
```
Note the error message is also very limited. It seems the stacktrace has been truncated
**To Reproduce**
Steps to reproduce the behavior:
1. Create a Hudi table
2. Configure Spark/Hadoop/Hudi classpath for CLI and start CLI with `hudi-cli-with-bundle.sh`
3. Connect to Hudi table with command `connect --path `
4. Run `cleans run`
**Expected behavior**
I believe there are 2 problems here:
- CLI bundle should be working
- When CLI fails, the stacktrace/error message shouldn't be truncated for debugging purposes
The second problem is the more concerning one to me now. I'd be happy to dive deeper and debug but currently I can't debug effectively due to limited error message
**Environment Description**
* Hudi version : 0.14.1 (I've tested with both EMR and OSS Hudi 0.14.1)
* Spark version : 3.5.0
* Hive version : 3.1.3
* Hadoop version : 3.3.6
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
See above
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the failure with hudi-cli-with-bundle.sh, the listed Spark/Hadoop/Hudi classpath, and the cleans run command after connecting to a table. Inspect how the CLI starts Spark and reports failures, focusing on why the org.apache.hudi.common.engine.HoodieEngineContext error and its stacktrace are truncated. Done means the bundle works for the reported flow and failures expose a useful full error message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, shell, spark
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100