aws / aws/sagemaker-python-sdk

Clean temporary files after loading query to dataframe

オープン
#5,100 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
component: feature store type: bug
主要言語
Python
スター
2.3k
フォーク
1.3k
平均マージ
1日 22時間
マージ済み PR(30日)
35

説明

**Describe the bug**
Loading data from feature group using `as_dataframe()` doesn't clean temporary `.csv files`

**To reproduce**
After creating feature group and ingesting data, load athena query and then use `as_dataframe()` method

**Expected behavior**
This methods loads .csv query file and returns the data inside, but it doesn't clean the .csv file itself.
So after running multiple queries it's really easy to just fill your local disk space up to maximum, because of multiple .csv files with query results

**Screenshots or logs**
`

def as_dataframe(self, **kwargs) -> DataFrame:
"""Download the result of the current query and load it into a DataFrame.

Args:
**kwargs (object): key arguments used for the method pandas.read_csv to be able to
have a better tuning on data. For more info read:
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html

Returns:
A pandas DataFrame contains the query result.
"""
query_state = self.get_query_execution().get("QueryExecution").get("Status").get("State")
if query_state != "SUCCEEDED":
if query_state in ("QUEUED", "RUNNING"):
raise RuntimeError(
f"Current query {self._current_query_execution_id} is still being executed."
)
raise RuntimeError(f"Failed to execute query {self._current_query_execution_id}")

output_filename = os.path.join(
tempfile.gettempdir(), f"{self._current_query_execution_id}.csv"
)
self.sagemaker_session.download_athena_query_result(
bucket=self._result_bucket,
prefix=self._result_file_prefix,
query_execution_id=self._current_query_execution_id,
filename=output_filename,
)

kwargs.pop("delimiter", None)
return pd.read_csv(filepath_or_buffer=output_filename, delimiter=",", **kwargs)

`

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start from the as_dataframe() entry point shown in the issue and trace the temporary CSV path through download_athena_query_result and pandas.read_csv. Verify that the query result still loads successfully and that the temporary CSV is removed after the method finishes.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
aws, pandas, python
領域
cloud, data
issue の種類
バグ
難易度
2/5
見積もり時間
1〜3時間
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
48/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。