apache / apache/hudi

Corrupted parquet file in hudi partition | Deletion of partition in Hudi

Open
#11,371 6 comments 0 reactions 0 assignees View on GitHub
issue:data-corruption
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

We are facing issue with a corrupted parquet file in one of the Hudi partition while reading Hudi table.

We are following the below steps
1. Shut down Hudi Writer to stop the ingestion
2. Read the Hudi Dataset and write with .option("hoodie.datasource.write.operation", "delete_partition")
3. val s3path = "s3://hudi-dev-use1/hudi_table/"
val data = spark.read.format("org.apache.hudi").load(s3path)
val deleteKeysDF = data.filter($"NAME" === "name_1").select("ID")
val writeStatus = deleteKeysDF.write.format("org.apache.hudi").option("hoodie.table.name", "hudi_table").
option("hoodie.datasource.write.recordkey.field", "ID").option("hoodie.datasource.write.table.name", "hudi_table").
option("hoodie.datasource.write.operation", "delete").option("hoodie.upsert.shuffle.parallelism", "2").mode(SaveMode.Append).save(s3path)
3. Bring back the Hudi Writer to continue the ingestion.

If we follow these steps, will we come out of the corrected parquet file issue?
Please share any other ways to completely remove hudi partition from Hudi Table.
Its a production issue, Any quick help is highly appreciated.

A clear and concise description of what you expected to happen.

PROD
* EMR -6.7.0
* Hudi version :0.11.1

* Spark version : 3.2.1

* Hadoop version : 3.2.1

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) :no

**Additional context**

Add any other context about the problem here.

**Stacktrace**

Can not read value at 0 in block -1 in file

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3655 in stage 13.0 failed 4 times, most recent failure: Lost task 3655.3 in stage 13.0 (TID 100513) (ip-100-67-247-116.8043.aws-int.thomsonreuters.com executor 24): java.lang.NegativeArraySizeException

at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:285)
at org.apache.parquet.bytes.BytesInput.toByteBuffer(BytesInput.java:237)
at org.apache.parquet.bytes.BytesInput.toInputStream(BytesInput.java:246)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readPageV1(VectorizedColumnReader.java:301)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.access$000(VectorizedColumnReader.java:47)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader$1.visit(VectorizedColumnReader.java:240)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader$1.visit(VectorizedColumnReader.java:236)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reported Spark parquet entry points, including VectorizedColumnReader and BytesInput, and compare them with the Hudi delete_partition and delete operations described in the issue. Determine whether the proposed sequence addresses the corrupted partition and define a safe, verified recovery procedure for the S3 table; the issue provides no repository files or tests to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.