[SUPPORT] org.apache.hudi.exception.HoodieIOException: Unable to create
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
The error message in the log is not specific enough.
```
org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:336)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:251)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:915)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:915)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
org.apache.hudi.exception.HoodieIOException: Unable to create :/data61/yarnenv/local/usercache/test/appcache/application_1686131065976_125148857/hudi-BITCASK-b2edc4fb-7dbd-42b9-9af8-a49a2da39889
at org.apache.hudi.common.util.collection.ExternalSpillableMap.getDiskBasedMap(ExternalSpillableMap.java:121)
at org.apache.hudi.common.util.collection.ExternalSpillableMap.getDiskBasedMapNumEntries(ExternalSpillableMap.java:140)
at org.apache.hudi.io.HoodieMergeHandle.init(HoodieMergeHandle.java:259)
at org.apache.hudi.io.HoodieMergeHandle.(HoodieMergeHandle.java:129)
at org.apache.hudi.io.HoodieMergeHandle.(HoodieMergeHandle.java:121)
java.io.IOException: Unable to create :/data61/yarnenv/local/usercache/test/appcache/application_1686131065976_125148857/hudi-BITCASK-b2edc4fb-7dbd-42b9-9af8-a49a2da39889
at org.apache.hudi.common.util.FileIOUtils.mkdir(FileIOUtils.java:70)
at org.apache.hudi.common.util.collection.DiskMap.(DiskMap.java:55)
at org.apache.hudi.common.util.collection.BitCaskDiskMap.(BitCaskDiskMap.java:98)
at org.apache.hudi.common.util.collection.ExternalSpillableMap.getDiskBasedMap(ExternalSpillableMap.java:118)
```
```java
public static void mkdir(File directory) throws IOException {
if (!directory.exists()) {
directory.mkdirs();
}
if (!directory.isDirectory()) {
throw new IOException("Unable to create :" + directory);
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with FileIOUtils.java at mkdir, then trace the reported calls through DiskMap and ExternalSpillableMap. Determine what context is missing from the “Unable to create” message and make the failure report specific; done means the reported error identifies the relevant cause or path clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100