GoogleCloudPlatform / GoogleCloudPlatform/gsutil

Add ability to print details for each error when running in `-m` mode without debug mode (`-D`)

Ouverte
#653 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Feature Request
Langage dominant
Python
Étoiles
918
Forks
335
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

Internal context for Google devs: I'm filing this to mirror the discussion in https://issuetracker.google.com/issues/126268394.

There are a variety of use cases in which this functionality would be helpful. As an example: When performing an object deletion, gsutil may receive a 503 response or hit a `socket.error` exception if the request hits the GCS backend, succeeds, but then *something bad* happens before the 200 response can reach the client. In this case, gsutil retries the request, then gets back a 404 for the retry because the object has already been deleted. From gsutil's current perspective, we should report all permanent failures to the user and treat them as errors. However, when looking at this and considering idempotency, we might want to treat 404s as OK, as they don't result in a change in the state of any buckets, objects, etc. In absence of some notion of idempotent requests at the GCS service level, the next best thing would be to either allow an option for gsutil to treat 404s as OK for certain operations (e.g. resource deletions), or to print out exception details for permanent failures so that we can manually examine them (and maybe determine that they were acceptable, as in the case of the above-mentioned 404). This feature request is for the latter.

Currently, when we haven't told gsutil to continue upon hitting an error (e.g. using the top-level `-m` option, `cp`'s `-c` option, etc.), it will print out a stack trace for the first permanent failure it encounters, then exit... but when running in parallel mode (using `-m`), the exceptions are suppressed and summarized at the end (e.g. "X files/objects could not be removed") - this FR is asking for an option that could be supplied along with `-m` that would make gsutil print out details about all the permanent failures it encounters (probably as they're encountered, rather than at the end). Note that while this is already achievable by using the top-level `-D` option, that also dumps a LOT of extra log entries to the console, most of which are probably uninteresting in the majority of cases.

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Commencez par suivre le traitement de -m au niveau supérieur dans gsutil et la manière dont les échecs permanents sont actuellement récapitulés, puis comparez ce chemin avec la sortie de débogage -D existante. Définissez le comportement de la nouvelle option pour afficher les détails de tous les échecs permanents sans la journalisation de débogage supplémentaire, notamment le moment où ces détails apparaissent. La tâche est terminée lorsque le comportement est implémenté et vérifié pour les opérations parallèles et des échecs représentatifs tels que des erreurs 503/de socket suivies d'erreurs 404.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
gcp, python
Domaine
cli, cloud
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
À l'abandon
Clarté
À clarifier
Accessibilité débutants
25/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.