GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil rsync fails with `Caught non-retryable exception - aborting rsync`
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
We have some files in GCS that we need to have them synchronized with S3, in AWS, but when I run `gsutil rsync`, it fails with `Caught non-retryable exception - aborting rsync`.
What we need to do is to run rsync every day so any files that has changed in GCS gets synchronized with AWS. When I run the command for the first time it copies the files correctly, but when I run it next time it fails.
So, this is the error I get when I run the command bucket vs bucket. Then I debugged it (with -D), and it looks like it fails on a specific file, which is directly in the bucket (no in a folder). The debug error message is the following:
```
crc32c: 'YW2ZwA=='
generation: 1605913220800799
md5Hash: 'uDEOva2LJsCKzE7U6vv0DQ=='
name: 'file.jar'
size: 10903598
timeCreated: datetime.datetime(2020, 11, 20, 23, 0, 20, 800000, tzinfo=)>]
prefixes: []>
DEBUG: Exception stack trace:
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 639, in _RunNamedCommandAndHandleExceptions
user_project=user_project)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 410, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1674, in RunCommand
diff_iterator = _DiffIterator(self, src_url, dst_url)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1007, in __init__
raise CommandException('Caught non-retryable exception - aborting rsync')
gslib.exception.CommandException: CommandException: Caught non-retryable exception - aborting rsync
CommandException: Caught non-retryable exception - aborting rsync
```
What I did next is to run the commnd agaist that specific file, and this is the error I get:
```
items: []
prefixes: []>
DEBUG: Exception stack trace:
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 639, in _RunNamedCommandAndHandleExceptions
user_project=user_project)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 410, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1642, in RunCommand
src_url = self._InsistContainer(self.args[0], False)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1625, in _InsistContainer
'slash to specify the directory.' % url_str)
gslib.exception.CommandException: CommandException: arg (gs://BUCKET-NAME/FILE.jar) does not name a directory, bucket, or bucket subdir.
If there is an object with the same path, please add a trailing
slash to specify the directory.
CommandException: arg (gs://BUCKET-NAME/FILE.jar) does not name a directory, bucket, or bucket subdir.
If there is an object with the same path, please add a trailing
slash to specify the directory.
```
The file name is `SOMETHING-SOMETHING-2.0.0-SNAPSHOT.jar`, so there are no special characters.
gsutil version: 4.65
boto version: 2.49.0
python: 3.7.3
I am not being able to go any further.
Contributor guide
Research direction
Start with gslib/commands/rsync.py, especially _DiffIterator and _InsistContainer, using the bucket-to-bucket and single-object commands described in the report. Reproduce the failure with gsutil 4.65 and the supplied GCS object, then trace why the existing object causes the non-retryable exception. Done means repeated GCS-to-S3 synchronization handles the reported root-level file without aborting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, gcp, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100