GoogleCloudPlatform / GoogleCloudPlatform/gsutil

Add ability to print details for each error when running in `-m` mode without debug mode (`-D`)

オープン
#653 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
Feature Request
主要言語
Python
スター
918
フォーク
335
PR マージ指標
30日以内にマージされた PR はありません

説明

Internal context for Google devs: I'm filing this to mirror the discussion in https://issuetracker.google.com/issues/126268394.

There are a variety of use cases in which this functionality would be helpful. As an example: When performing an object deletion, gsutil may receive a 503 response or hit a `socket.error` exception if the request hits the GCS backend, succeeds, but then *something bad* happens before the 200 response can reach the client. In this case, gsutil retries the request, then gets back a 404 for the retry because the object has already been deleted. From gsutil's current perspective, we should report all permanent failures to the user and treat them as errors. However, when looking at this and considering idempotency, we might want to treat 404s as OK, as they don't result in a change in the state of any buckets, objects, etc. In absence of some notion of idempotent requests at the GCS service level, the next best thing would be to either allow an option for gsutil to treat 404s as OK for certain operations (e.g. resource deletions), or to print out exception details for permanent failures so that we can manually examine them (and maybe determine that they were acceptable, as in the case of the above-mentioned 404). This feature request is for the latter.

Currently, when we haven't told gsutil to continue upon hitting an error (e.g. using the top-level `-m` option, `cp`'s `-c` option, etc.), it will print out a stack trace for the first permanent failure it encounters, then exit... but when running in parallel mode (using `-m`), the exceptions are suppressed and summarized at the end (e.g. "X files/objects could not be removed") - this FR is asking for an option that could be supplied along with `-m` that would make gsutil print out details about all the permanent failures it encounters (probably as they're encountered, rather than at the end). Note that while this is already achievable by using the top-level `-D` option, that also dumps a LOT of extra log entries to the console, most of which are probably uninteresting in the majority of cases.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

まず、gsutil のトップレベルでの -m の処理と、恒久的な失敗が現在どのように要約されているかを追跡し、その後、その経路を既存の -D デバッグ出力と比較します。追加のデバッグロギングなしで、すべての恒久的な失敗の詳細を出力する新しいオプションの動作を、そうした詳細がいつ表示されるかも含めて定義します。並列操作と、503/ソケットエラーに続いて404エラーが発生するケースなどの代表的な失敗について、その動作が実装され検証されれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
gcp, python
領域
cli, cloud
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
説明が足りない
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。