apache / apache/iceberg-go

Encryption Support

Open
#1,289 11 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
463
Forks
232
Avg merge
2d 11h
Merged PRs (30d)
121

Description

### Feature Request / Improvement

iceberg-go will read the `encryption-keys` metadata, but it doesn't support encrypting / decrypting data using that metadata.

I think the biggest part of this will be figuring out the proper interfaces. There will many different implementations of these interfaces over time.

I'd love to hear everyone's idea on the basic design here:

# Interfaces

- [ ] Encryption interfaces + Plaintext manager — EncryptionManager, EncryptedInputFile/EncryptedOutputFile, EncryptionKeyMetadata, NativeEncryption*, and the no-op PlaintextEncryptionManager.
- [ ] KMS client interface — KeyManagementClient (wrapKey/unwrapKey/supportsKeyGeneration/generateKey), plus a test/in-memory impl and encryption.kms-type/encryption.kms-impl catalog-property wiring.
- [ ] StandardEncryptionManager — envelope KEK/DEK logic, local DEK generation, key cache, rotation/timestamp tracking, encryption.key-id + encryption.data-key-length table properties. Consumes the existing encryption-keys metadata array.

# Integration
- [ ] EncryptingFileIO wrapper — slot encryption into the io.IO layer: decrypt on Open of Data/Manifest/ManifestList/Statistics files based on key_metadata, encrypt on write.
- [ ] Parquet native encryption for data/delete files — wire FileEncryptionProperties/FileDecryptionProperties from NativeEncryptionOutputFile into table/internal/parquet_files.go (write) and the scan read path.
- [ ] Manifest + manifest-list encryption — whole-file AES-GCM using the existing key_metadata fields, including the manifest-list double-encryption (KEK + timestamp AAD) on the write/read planners.
- [ ] Puffin blob / statistics-file encryption — encrypt DV and stats blobs; populate StatisticsFile.KeyMetadata.

# Testing
- [ ] Format-version compatibility + end-to-end + cross-client tests — reject encryption props on v1, gate behavior, and round-trip a Java-produced encrypted table end-to-end (the equivalent of the DV cross-client compliance PR #1131).

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing encryption-keys metadata handling and the io.IO layer, then inspect table/internal/parquet_files.go and the scan read path. Map the proposed encryption interfaces, KMS wiring, file-format integrations, and format-version behavior before implementing. Done means encrypted tables can be read and written across the listed file types with end-to-end and cross-client coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.