GoogleCloudPlatform / GoogleCloudPlatform/gsutil

Add ability to print details for each error when running in `-m` mode without debug mode (`-D`)

Offen
#653 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Feature Request
Vorherrschende Sprache
Python
Sterne
918
Forks
335
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

Internal context for Google devs: I'm filing this to mirror the discussion in https://issuetracker.google.com/issues/126268394.

There are a variety of use cases in which this functionality would be helpful. As an example: When performing an object deletion, gsutil may receive a 503 response or hit a `socket.error` exception if the request hits the GCS backend, succeeds, but then *something bad* happens before the 200 response can reach the client. In this case, gsutil retries the request, then gets back a 404 for the retry because the object has already been deleted. From gsutil's current perspective, we should report all permanent failures to the user and treat them as errors. However, when looking at this and considering idempotency, we might want to treat 404s as OK, as they don't result in a change in the state of any buckets, objects, etc. In absence of some notion of idempotent requests at the GCS service level, the next best thing would be to either allow an option for gsutil to treat 404s as OK for certain operations (e.g. resource deletions), or to print out exception details for permanent failures so that we can manually examine them (and maybe determine that they were acceptable, as in the case of the above-mentioned 404). This feature request is for the latter.

Currently, when we haven't told gsutil to continue upon hitting an error (e.g. using the top-level `-m` option, `cp`'s `-c` option, etc.), it will print out a stack trace for the first permanent failure it encounters, then exit... but when running in parallel mode (using `-m`), the exceptions are suppressed and summarized at the end (e.g. "X files/objects could not be removed") - this FR is asking for an option that could be supplied along with `-m` that would make gsutil print out details about all the permanent failures it encounters (probably as they're encountered, rather than at the end). Note that while this is already achievable by using the top-level `-D` option, that also dumps a LOT of extra log entries to the console, most of which are probably uninteresting in the majority of cases.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start by tracing gsutil's top-level -m handling and how permanent failures are currently summarized, then compare that path with the existing -D debug output. Define the new option's behavior for printing details for all permanent failures without the extra debug logging, including when those details appear. Done means the behavior is implemented and verified for parallel operations and representative failures such as 503/socket errors followed by 404s.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
gcp, python
Bereich
cli, cloud
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.