apache / apache/parquet-format

Support for GDPR erase requirements

Open
#341 4 comments 0 reactions 0 assignees View on GitHub
Priority: Major Type: enhancement
Dominant language
Thrift
Stars
2.6k
Forks
508
Avg merge
2d 8h
Merged PRs (30d)
4

Description

As understand it Parquet is a write once thing. So mutating data inside Parquet files is not an option. Now there is a new cross EU law coming in effect May 2018 that requires companies to delete data pertaining a customer if being asked to do so.

Our case is quite simple, our biggest parquet tables collect 7.5 billion rows a month. So removing data by duplicating this table whilst filtering out the unwanted customer data is not feasible.

Perhaps there is some way to remove particular data? Or perhaps there is an efficient way to do read/filter/write? Perhaps zeroing the data is an idea to not change the layout of the files.

Not sure if this is the right platform to start this discussion but I think more people will have this issue once it becomes clear that data needs to be deleted in all places, also in parquet files. Companies fase multi million dollar fines if they don't comply with GDPR.

**Reporter**: [Machiel Groeneveld](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=machielg)

**Note**: *This issue was originally created as [PARQUET-1155](https://issues.apache.org/jira/browse/PARQUET-1155). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or implementation entry point. Start by reviewing the GDPR erase requirement and the migration documentation for PARQUET-1155, then establish the supported behavior and acceptance criteria before implementation.

Written by the indexing model from the issue text.

Assessment

Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.