GoogleCloudPlatform / GoogleCloudPlatform/gsutil

Rsync Race Condition

Open
#1,336 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Hi,

I think I'm experiencing a race condition when using rsync between a bucket and local disk (also possibly between two buckets).

If a file is deleted from the bucket after the _DiffIterator object is created I am seeing the following:

`NotFoundException: 404 gs://xxxxxxxxxxxxxxxxx`

This bucket is pretty busy with read/write operations so it is likely a file can be deleted after the diff is generated.

Is this expected?

This is problematic as the script exits with a non-zero code and I'm trying to implement some functionality using Kubernetes CronJobs.

I'm using the following container: **google/cloud-sdk:351.0.0-slim**

Thanks

Contributor guide

Open the contributing guide

Research direction

Start with the gsutil rsync path around the mentioned _DiffIterator and reproduce the deletion-after-diff case using google/cloud-sdk:351.0.0-slim. Determine the expected behavior when a bucket object disappears and verify the resulting exit status for a Kubernetes CronJob workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, kubernetes, python
Domain
cli, cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.