apache / apache/datafusion-python

Expose object_store for direct use

未關閉
#1,008 6 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
Python
星號
604
分支
174
平均合併
2 天 22 小時
30 天內合併 PR
5

描述

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I need to be able to delete old resources generated by `write_parquet()` and similar methods, move them out of the way, or do other such operations that broadly fall in the category of 'data/artifact cleanup'. Datafusion doesn't directly implement such move/delete operations, so this requires a different library. Depending on what environment i'm operating in (local file system, S3, google bucket) this requires a slightly different setup and operation.

However, datafusion ships with `ObjectStore`, a generic frontend for many different object store systems, mainly S3 and its equivalents in other cloud environments, but also providing an api-compatible local file storage version of this. This is used to be able to read and write to such object stores from within datafusion. In datafusion-python, these ObjectStore objects are opaque handles that are only useful for registering with a session context. In rust however, these also allow the user to directly manipulate objects in these stores, fetch them, delete them, move them, etc.

**Describe the solution you'd like**
I would like the datafusion-python version of ObjectStore to not just be an opaque handle, but instead allow access to the underlying methods. this will allow me to generically implement operations on generated artifacts that are not doable in datafusion directly.

**Describe alternatives you've considered**
The workaround is to use an s3-compatible library directly. This doesn't help with local files though, which still requires a separate code path.

Another possibility is to have a separate python library wrapping the rust object_store crate, as arguably it's not the job of datafusion-python to provide a good API for this. However, it's useful to be able to define just one ObjectStore (like from a configuration) and use it both for datafusion and for related object-store operations like artifact cleanup.

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

先檢查 issue 中所描述的 datafusion-python ObjectStore binding,以及底層的 Rust object_store API。確定需要公開哪些直接物件操作,並驗證產生的 Python 介面能夠用於本機儲存與雲端後端儲存;完成的標準是使用者可以重複使用已設定的 ObjectStore,既用於清理 artifact,也用於存取 DataFusion。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python, rust
領域
backend-api-design, data-engineering
Issue 類型
功能
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。