GoogleCloudPlatform / GoogleCloudPlatform/gsutil
cp -L does not report failed files in the log
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
I ran a command like this:
```
gsutil -m cp -L cp.log -r -n data/* gs://
```
At the end of the output gsutil reported that some files failed to copy:
```
CommandException: 462 files/objects could not be transferred.
```
However when I look in cp.log there are no files with a failed status.
The gsutil output did show a number of errors (exactly 462 of them) like the following, but the error messages don't appear to include the name of the relevant file so I can't figure out which files failed to transfer:
```
'utf-8' codec can't encode character '\udca9' in position 76: surrogates not allowed
```
The gsutil output also included the following estimate for the number of files that it would copy:
```
Estimated work for this command: objects: 1340188, total size: 26.85 TiB
```
And running `wc -l cp.log` shows the following - exactly 1 line for every file except the 462 failed ones, plus 1 header line:
```
# wc -l cp.log
1339727 cp.log
```
Which suggests that the files that failed did not get included in the log file.
This appears to be a bug with cp's log behavior, but also is there any workaround to find out which files failed to copy? Thanks
Contributor guide
Research direction
No repository files or tests are named. Start by reproducing the gsutil -m cp -L command and trace the cp logging path around failed transfers and UTF-8 encoding errors. Done means every failed file is represented in cp.log with an identifiable path, with coverage for this failure case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100