GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil : OSError, Not a directory, can't trace, debug, fix

Open
#1,058 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

@sergeykish has opend an issue in [python-storage](https://github.com/googleapis/python-storage/issues/181) repo , but it's related to gsutil so I am opening the issue here for better response.

**Environment details**

$ uname -a
Linux notebook 5.7.2-arch1-1 googleapis/google-cloud-python#1 SMP PREEMPT Wed, 10 Jun 2020 20:36:24 +0000 x86_64 GNU/Linux
$ python --version
Python 3.8.3
$ cd /opt/google-cloud-sdk/
$ cat VERSION
293.0.0

**Steps to reproduce**

1. make foo object and foo path to the object
2. rsync to local folder
OSError: Not a directory

I know I can't make file and directory with same name in directory. I'd like to make it encode node type in name or maybe skip it. Unfortunately I can't trace where it throws error. I've tried python -mpdb ..., it does not catch. Does it spawn another process? How can I debug it?

**Code example**

$ echo foo > foo
$ echo bar > bar
$ gsutil cp foo gs://sergeykish.com/foo
$ gsutil cp bar gs://sergeykish.com/foo/bar
$ gsutil rsync -r gs://sergeykish.com/ .

WARNING: gsutil rsync uses hashes when modification time is not available at
both the source and destination. Your crcmod installation isn't using the
module's C extension, so checksumming will run very slowly. If this is your
first rsync since updating gsutil, this rsync can take significantly longer than
usual. For help installing the extension, please see "gsutil help crcmod".

Building synchronization state...
Starting synchronization...
Copying gs://sergeykish.com/foo...
Copying gs://sergeykish.com/foo/bar...
OSError: Not a directory. 8.0 B]

**Stack trace**

No stack trace.

PDB traces to Exec

(Pdb) import sys
(Pdb) old_sys_exit = sys.exit
(Pdb) def new_sys_exit (value) : breakpoint()
(Pdb) sys.exit = new_sys_exit
(Pdb) b new_sys_exit
End of file
(Pdb) c
...
OSError: Not a directory..6 KiB]
--Return--
> (1)new_sys_exit()->None
(Pdb) where
...
-> """A convenience wrapper for starting gsutil."""
/opt/google-cloud-sdk/bin/bootstrapping/gsutil.py(102)main()
-> bootstrapping.ExecutePythonTool('platform/gsutil', 'gsutil', *args)
/opt/google-cloud-sdk/bin/bootstrapping/bootstrapping.py(107)ExecutePythonTool()
-> _ExecuteTool(
/opt/google-cloud-sdk/bin/bootstrapping/bootstrapping.py(199)_ExecuteTool()
-> execution_utils.Exec(args + sys.argv[1:], env=_GetToolEnv())
/opt/google-cloud-sdk/lib/googlecloudsdk/core/execution_utils.py(357)Exec()
-> sys.exit(ret_val)
> (1)new_sys_exit()->None

Contributor guide

Open the contributing guide

Research direction

Reproduce the collision with `gsutil rsync -r` using the commands in the issue, then trace the copy path from `/opt/google-cloud-sdk/bin/bootstrapping/gsutil.py` through `bootstrapping.py` and `execution_utils.py`. Determine where the object/path conflict becomes `OSError: Not a directory`; done should be a decided and tested behavior for this case without that unhandled error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.