apache / apache/iceberg-python
Option to specify SSE-KMS or SSE-S3 encryption when writing data with load_catalog / append
- 主要語言
- Python
- 星號
- 1.1k
- 分支
- 588
- 平均合併
- 1 天 23 小時
- 30 天內合併 PR
- 84
描述
### Question
Hello team,
I am using pyiceberg to load data into an Iceberg table stored in Amazon S3.
While doing this, I am facing an explicit deny from an AWS Service Control Policy (SCP) that blocks multipart uploads without encryption. I cannot modify the SCP.
Error excerpt:
`
OSError: When initiating multiple part upload for key 'iceberg/DEV/dataset/test_4_matdoc/metadata/...'
in bucket 'pt-s3-project-bucketname':
AWS Error ACCESS_DENIED during CreateMultipartUpload operation:
User: arn:aws:sts::... is not authorized to perform: s3:PutObject
with an explicit deny in a service control policy
`
This happens during calls like:
`def load_iceberg_table(table, arrow_table):
catalog = load_catalog("glue", **{"type": "glue"})
iceberg_table: Table = catalog.load_table(f"{DATABASE}.{table}")
try:
logger.info("Appending data to Iceberg table...")
iceberg_table.append(df=arrow_table)
logger.info("Successfully appended data to Iceberg table.")
except ClientError as e:
logger.error(f"Iceberg append ClientError: {e}")
raise
except Exception as e:
logger.error(f"Unexpected Iceberg error: {e}")
raise`
From my understanding, pyiceberg uses S3 multipart upload under the hood, but I haven’t found a documented way to configure SSE-KMS or SSE-S3 parameters for these writes.
Question:
Is there currently a way to pass S3 upload parameters (like ServerSideEncryption, SSEKMSKeyId) via load_catalog, append, or FileIO configuration?
If not, could this be added as a feature so that environments with encryption-required SCPs can still use pyiceberg without policy changes?
Thanks!
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
首先追蹤 load_catalog 和 append 如何到達 FileIO S3 multipart 上傳路徑,並使用回報的 ACCESS_DENIED 案例作為重現案例。檢查是否能透過 catalog、append 或 FileIO 設定公開加密參數。完成的標準是:可以為寫入設定所要求的 SSE-S3 或 SSE-KMS 設定,並且已驗證加密上傳路徑。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- aws, python
- 領域
- cloud, security
- Issue 類型
- 功能
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 冷清
- 描述清晰度
- 基本清楚
- 新手友好度
- 50/100