GoogleCloudPlatform / GoogleCloudPlatform/gsutil
Add ability to print details for each error when running in `-m` mode without debug mode (`-D`)
- 主要言語
- Python
- スター
- 918
- フォーク
- 335
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
Internal context for Google devs: I'm filing this to mirror the discussion in https://issuetracker.google.com/issues/126268394.
There are a variety of use cases in which this functionality would be helpful. As an example: When performing an object deletion, gsutil may receive a 503 response or hit a `socket.error` exception if the request hits the GCS backend, succeeds, but then *something bad* happens before the 200 response can reach the client. In this case, gsutil retries the request, then gets back a 404 for the retry because the object has already been deleted. From gsutil's current perspective, we should report all permanent failures to the user and treat them as errors. However, when looking at this and considering idempotency, we might want to treat 404s as OK, as they don't result in a change in the state of any buckets, objects, etc. In absence of some notion of idempotent requests at the GCS service level, the next best thing would be to either allow an option for gsutil to treat 404s as OK for certain operations (e.g. resource deletions), or to print out exception details for permanent failures so that we can manually examine them (and maybe determine that they were acceptable, as in the case of the above-mentioned 404). This feature request is for the latter.
Currently, when we haven't told gsutil to continue upon hitting an error (e.g. using the top-level `-m` option, `cp`'s `-c` option, etc.), it will print out a stack trace for the first permanent failure it encounters, then exit... but when running in parallel mode (using `-m`), the exceptions are suppressed and summarized at the end (e.g. "X files/objects could not be removed") - this FR is asking for an option that could be supplied along with `-m` that would make gsutil print out details about all the permanent failures it encounters (probably as they're encountered, rather than at the end). Note that while this is already achievable by using the top-level `-D` option, that also dumps a LOT of extra log entries to the console, most of which are probably uninteresting in the majority of cases.
コントリビューションガイド
調査の方向性
まず、gsutil のトップレベルでの -m の処理と、恒久的な失敗が現在どのように要約されているかを追跡し、その後、その経路を既存の -D デバッグ出力と比較します。追加のデバッグロギングなしで、すべての恒久的な失敗の詳細を出力する新しいオプションの動作を、そうした詳細がいつ表示されるかも含めて定義します。並列操作と、503/ソケットエラーに続いて404エラーが発生するケースなどの代表的な失敗について、その動作が実装され検証されれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- gcp, python
- 領域
- cli, cloud
- issue の種類
- 機能追加
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 活発さ
- 停滞
- 明瞭さ
- 説明が足りない
- 初心者へのやさしさ
- 25/100