Corrupted parquet file in hudi partition | Deletion of partition in Hudi
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
We are facing issue with a corrupted parquet file in one of the Hudi partition while reading Hudi table.
We are following the below steps
1. Shut down Hudi Writer to stop the ingestion
2. Read the Hudi Dataset and write with .option("hoodie.datasource.write.operation", "delete_partition")
3. val s3path = "s3://hudi-dev-use1/hudi_table/"
val data = spark.read.format("org.apache.hudi").load(s3path)
val deleteKeysDF = data.filter($"NAME" === "name_1").select("ID")
val writeStatus = deleteKeysDF.write.format("org.apache.hudi").option("hoodie.table.name", "hudi_table").
option("hoodie.datasource.write.recordkey.field", "ID").option("hoodie.datasource.write.table.name", "hudi_table").
option("hoodie.datasource.write.operation", "delete").option("hoodie.upsert.shuffle.parallelism", "2").mode(SaveMode.Append).save(s3path)
3. Bring back the Hudi Writer to continue the ingestion.
If we follow these steps, will we come out of the corrected parquet file issue?
Please share any other ways to completely remove hudi partition from Hudi Table.
Its a production issue, Any quick help is highly appreciated.
A clear and concise description of what you expected to happen.
PROD
* EMR -6.7.0
* Hudi version :0.11.1
* Spark version : 3.2.1
* Hadoop version : 3.2.1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) :no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
Can not read value at 0 in block -1 in file
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3655 in stage 13.0 failed 4 times, most recent failure: Lost task 3655.3 in stage 13.0 (TID 100513) (ip-100-67-247-116.8043.aws-int.thomsonreuters.com executor 24): java.lang.NegativeArraySizeException
at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:285)
at org.apache.parquet.bytes.BytesInput.toByteBuffer(BytesInput.java:237)
at org.apache.parquet.bytes.BytesInput.toInputStream(BytesInput.java:246)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readPageV1(VectorizedColumnReader.java:301)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.access$000(VectorizedColumnReader.java:47)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader$1.visit(VectorizedColumnReader.java:240)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader$1.visit(VectorizedColumnReader.java:236)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the reported Spark parquet entry points, including VectorizedColumnReader and BytesInput, and compare them with the Hudi delete_partition and delete operations described in the issue. Determine whether the proposed sequence addresses the corrupted partition and define a safe, verified recovery procedure for the S3 table; the issue provides no repository files or tests to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100