apache / apache/hudi

GDPR: Support deletions of records on all versions of Hudi dataset

オープン
#14,653 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
area:table-service from-jira priority:high type:improvement
主要言語
Java
スター
6.2k
フォーク
2.5k
平均マージ
2日 4時間
マージ済み PR(30日)
112

説明

Incremental Pull should also stop returning the record on historical  datset when we delete them from latest snapshot.

 

Context from Mailing list email :

 

Hello,

I am Siva's colleague and I am working on the problem below as well.

I would like to describe what we are trying to achieve with Hudi as well as our current way of working and our GDPR and "Right To Be Forgotten " compliance policies.

Our requirements :
- We wish to apply a strict interpretation of the RTBF.  In other words, when we remove a person's data, it should be throughout the historical data and not just the latest snapshot.
- We wish to use Hudi to reduce our storage requirements using upserts and don't want to have duplicates between commits.
- We wish to retain history for persons who have not requested to be forgotten and therefore we do not want to delete commit files from the history as some have proposed.

We have tried a couple of solutions, but so far without success :
- replay the data omitting the data of the persons who have requested to be forgotten.  We wanted to manipulate the commit times to rebuild the history.
We found that we couldn't manipulate the commit times and retain the history.

- replay the data omitting the data of the persons who have requested to be forgotten, but writing to a date-based partition folder using the "partitionpath" parameter.
We found that commits using upserts between the partitionpath folders, do not ignore data that is unchanged between 2 commit dates as when using the default commit file system, so we will not save on our storage or speed up our  processing using this technique.

So basically we would like to find a way to apply a strict RTBF, GDPR, maintain history and time-travel (large history) and save storage space using Hudi.

Can anyone see a way to achieve this?

Kind Regards,
David Rosalia

 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1212
- Type: Improvement
- Affects version(s):
- 0.9.0

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

この issue には、ソースファイル、テスト、コードのエントリーポイントが記載されていません。まず、履歴データセットに対する Hudi の増分取得の動作を追跡し、最新の Snapshot からの削除がどのように表現されるかを明確にしてください。履歴データ全体にわたって削除されたレコードが返されず、無関係な履歴と Time-Travel は引き続き利用できれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
data-engineering
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
説明が足りない
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。