apache / apache/hudi

[SUPPORT] org.apache.hudi.exception.HoodieIOException: Unable to create

Open
#9,154 4 comments 0 reactions 0 assignees View on GitHub
area:writer issue:stability priority:medium
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

The error message in the log is not specific enough.

```
org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:336)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:251)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:915)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:915)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)

org.apache.hudi.exception.HoodieIOException: Unable to create :/data61/yarnenv/local/usercache/test/appcache/application_1686131065976_125148857/hudi-BITCASK-b2edc4fb-7dbd-42b9-9af8-a49a2da39889
at org.apache.hudi.common.util.collection.ExternalSpillableMap.getDiskBasedMap(ExternalSpillableMap.java:121)
at org.apache.hudi.common.util.collection.ExternalSpillableMap.getDiskBasedMapNumEntries(ExternalSpillableMap.java:140)
at org.apache.hudi.io.HoodieMergeHandle.init(HoodieMergeHandle.java:259)
at org.apache.hudi.io.HoodieMergeHandle.(HoodieMergeHandle.java:129)
at org.apache.hudi.io.HoodieMergeHandle.(HoodieMergeHandle.java:121)

java.io.IOException: Unable to create :/data61/yarnenv/local/usercache/test/appcache/application_1686131065976_125148857/hudi-BITCASK-b2edc4fb-7dbd-42b9-9af8-a49a2da39889
at org.apache.hudi.common.util.FileIOUtils.mkdir(FileIOUtils.java:70)
at org.apache.hudi.common.util.collection.DiskMap.(DiskMap.java:55)
at org.apache.hudi.common.util.collection.BitCaskDiskMap.(BitCaskDiskMap.java:98)
at org.apache.hudi.common.util.collection.ExternalSpillableMap.getDiskBasedMap(ExternalSpillableMap.java:118)
```

```java
public static void mkdir(File directory) throws IOException {
if (!directory.exists()) {
directory.mkdirs();
}

if (!directory.isDirectory()) {
throw new IOException("Unable to create :" + directory);
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with FileIOUtils.java at mkdir, then trace the reported calls through DiskMap and ExternalSpillableMap. Determine what context is missing from the “Unable to create” message and make the failure report specific; done means the reported error identifies the relevant cause or path clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.