GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil rsync fails with `Caught non-retryable exception - aborting rsync`

Open
#1,356 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

We have some files in GCS that we need to have them synchronized with S3, in AWS, but when I run `gsutil rsync`, it fails with `Caught non-retryable exception - aborting rsync`.

What we need to do is to run rsync every day so any files that has changed in GCS gets synchronized with AWS. When I run the command for the first time it copies the files correctly, but when I run it next time it fails.

So, this is the error I get when I run the command bucket vs bucket. Then I debugged it (with -D), and it looks like it fails on a specific file, which is directly in the bucket (no in a folder). The debug error message is the following:

```
crc32c: 'YW2ZwA=='
generation: 1605913220800799
md5Hash: 'uDEOva2LJsCKzE7U6vv0DQ=='
name: 'file.jar'
size: 10903598
timeCreated: datetime.datetime(2020, 11, 20, 23, 0, 20, 800000, tzinfo=)>]
prefixes: []>
DEBUG: Exception stack trace:
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 639, in _RunNamedCommandAndHandleExceptions
user_project=user_project)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 410, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1674, in RunCommand
diff_iterator = _DiffIterator(self, src_url, dst_url)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1007, in __init__
raise CommandException('Caught non-retryable exception - aborting rsync')
gslib.exception.CommandException: CommandException: Caught non-retryable exception - aborting rsync

CommandException: Caught non-retryable exception - aborting rsync
```

What I did next is to run the commnd agaist that specific file, and this is the error I get:

```
items: []
prefixes: []>
DEBUG: Exception stack trace:
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 639, in _RunNamedCommandAndHandleExceptions
user_project=user_project)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 410, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1642, in RunCommand
src_url = self._InsistContainer(self.args[0], False)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1625, in _InsistContainer
'slash to specify the directory.' % url_str)
gslib.exception.CommandException: CommandException: arg (gs://BUCKET-NAME/FILE.jar) does not name a directory, bucket, or bucket subdir.
If there is an object with the same path, please add a trailing
slash to specify the directory.

CommandException: arg (gs://BUCKET-NAME/FILE.jar) does not name a directory, bucket, or bucket subdir.
If there is an object with the same path, please add a trailing
slash to specify the directory.
```

The file name is `SOMETHING-SOMETHING-2.0.0-SNAPSHOT.jar`, so there are no special characters.

gsutil version: 4.65
boto version: 2.49.0
python: 3.7.3

I am not being able to go any further.

Contributor guide

Open the contributing guide

Research direction

Start with gslib/commands/rsync.py, especially _DiffIterator and _InsistContainer, using the bucket-to-bucket and single-object commands described in the report. Reproduce the failure with gsutil 4.65 and the supplied GCS object, then trace why the existing object causes the non-retryable exception. Done means repeated GCS-to-S3 synchronization handles the reported root-level file without aborting.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, gcp, python
Domain
cli, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.