GoogleCloudPlatform / GoogleCloudPlatform/gsutil
IOERROR: operation not permitted on rsync
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
I'm trying to use rsync to upload from a read-only filesystem on an external disk to a google cloud bucket. The process worked for a handful of folders, but after successfully uploding thousands of files has begun consistently failing for one of my directories (total contents: 40 GB). I tried switching to another similarly sized directory and it worked for a while but is now failing as well.
I've tried uninstalling the google cloud SDK but I get the same error. I've tried poking around in the files mentioned by the stack trace but haven't been able to find anything. This has been going on for a couple days, so it's not a rate limit issue.
I've upgraded my account to a full paid plan and have a card attached, so I don't think it's a payments issue. Is there some maximum bucket contents size setting somewhere that I've missed?
At the very least, a more descriptive error message would be really helpful. Thanks!
```
$ gsutil rsync -d -r /Volumes/ExtHD/my_directory/ gs://mike_personal_backups/my_directory
Building synchronization state...
Starting synchronization
Copying from named pipe...
Traceback (most recent call last):
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gsutil", line 22, in
gsutil.RunMain()
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gsutil.py", line 114, in RunMain
sys.exit(gslib.__main__.main())
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 381, in main
perf_trace_token=perf_trace_token, user_project=user_project)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 567, in _RunNamedCommandAndHandleExceptions
user_project=user_project)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 319, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1425, in RunCommand
fail_on_error=True, seek_ahead_iterator=seek_ahead_iterator)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1372, in Apply
arg_checker, should_return_results, fail_on_error)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1443, in _SequentialApply
worker_thread.PerformTask(task, self)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/command.py", line 2109, in PerformTask
results = task.func(cls, task.args, thread_state=self.thread_gsutil_api)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1215, in _RsyncFunc
preserve_posix=cls.preserve_posix_attrs)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/copy_helper.py", line 3306, in PerformCopy
allow_splitting=allow_splitting)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/copy_helper.py", line 1803, in _UploadFileToObject
dst_obj_metadata, preconditions, gsutil_api)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/copy_helper.py", line 1496, in _UploadFileToObjectNonResumable
fields=UPLOAD_RETURN_FIELDS)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/cloud_api_delegator.py", line 293, in UploadObjectStreaming
encryption_tuple=encryption_tuple, fields=fields)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1309, in UploadObjectStreaming
apitools_strategy=apitools_transfer.RESUMABLE_UPLOAD, total_size=None)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1158, in _UploadObject
additional_headers, progress_callback)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1229, in _PerformResumableUpload
additional_headers=addl_headers)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 930, in StreamInChunks
additional_headers=additional_headers)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 883, in __StreamMedia
additional_headers=additional_headers)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 983, in __SendChunk
self.stream, start, self.chunksize)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/buffered_stream.py", line 34, in __init__
self.__buffered_data = self.__stream.read(size)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/hashing_helper.py", line 412, in read
data = self._orig_fp.read(size)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/resumable_streaming_upload.py", line 128, in read
new_data = self._orig_fp.read(bytes_remaining)
IOError: [Errno 1] Operation not permitted
```
Now attempting the same operation on a different folder. It worked for thousands of files, but is now failing:
```
$ gsutil rsync -d -r /Volumes/ExtHD/my_other_directory/ gs://mike_personal_backups/my_other_directory
Building synchronization state...
At source listing 10000...
At source listing 20000...
At source listing 30000...
At source listing 40000...
At source listing 50000...
At source listing 60000...
Starting synchronization
Copying from named pipe...
Traceback (most recent call last):
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gsutil", line 22, in
gsutil.RunMain()
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gsutil.py", line 114, in RunMain
sys.exit(gslib.__main__.main())
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 381, in main
perf_trace_token=perf_trace_token, user_project=user_project)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 567, in _RunNamedCommandAndHandleExceptions
user_project=user_project)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 319, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1425, in RunCommand
fail_on_error=True, seek_ahead_iterator=seek_ahead_iterator)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1372, in Apply
arg_checker, should_return_results, fail_on_error)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1443, in _SequentialApply
worker_thread.PerformTask(task, self)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/command.py", line 2109, in PerformTask
results = task.func(cls, task.args, thread_state=self.thread_gsutil_api)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1215, in _RsyncFunc
preserve_posix=cls.preserve_posix_attrs)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/copy_helper.py", line 3306, in PerformCopy
allow_splitting=allow_splitting)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/copy_helper.py", line 1803, in _UploadFileToObject
dst_obj_metadata, preconditions, gsutil_api)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/copy_helper.py", line 1496, in _UploadFileToObjectNonResumable
fields=UPLOAD_RETURN_FIELDS)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/cloud_api_delegator.py", line 293, in UploadObjectStreaming
encryption_tuple=encryption_tuple, fields=fields)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1309, in UploadObjectStreaming
apitools_strategy=apitools_transfer.RESUMABLE_UPLOAD, total_size=None)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1158, in _UploadObject
additional_headers, progress_callback)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1229, in _PerformResumableUpload
additional_headers=addl_headers)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 930, in StreamInChunks
additional_headers=additional_headers)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 883, in __StreamMedia
additional_headers=additional_headers)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 983, in __SendChunk
self.stream, start, self.chunksize)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/buffered_stream.py", line 34, in __init__
self.__buffered_data = self.__stream.read(size)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/hashing_helper.py", line 412, in read
data = self._orig_fp.read(size)
File "/Users/delgadom/google-cloud-sdk/platform/gsutil/gslib/resumable_streaming_upload.py", line 128, in read
new_data = self._orig_fp.read(bytes_remaining)
IOError: [Errno 1] Operation not permitted
```
Contributor guide
Assessment
This issue has not been assessed yet.