vmware / vmware/pyvmomi

WaitForUpdates stuck for more than 2 days

Open
#548 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs verification
Dominant language
Python
Stars
2.3k
Forks
763
PR merge metrics
No merged PRs in 30d

Description

[pyvmomi version = 6.0.0.2016.6]
I'm facing one issue while creating snapshot. The api got stuck for more than 2 days or in some cases until i kill the process. I did some analysis here and need your opinion on that. Let's start with thread dump :
File: "pyVmomi/VmomiSupport.py", line 560, in call
File: "pyVmomi/VmomiSupport.py", line 383, in _InvokeAccessor
File: "pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor
File: "pyVmomi/VmomiSupport.py", line 566, in
File: "pyVmomi/VmomiSupport.py", line 375, in _InvokeMethod
File: "pyVmomi/SoapAdapter.py", line 1282, in InvokeMethod
File: "pyVmomi/SoapAdapter.py", line 783, in Deserialize
File: "pyVmomi/SoapAdapter.py", line 456, in ReadDocument
File: "httplib.py", line 522, in read
File: "httplib.py", line 569, in _read_chunked
File: "httplib.py", line 619, in _safe_read
File: "socket.py", line 377, in read
File: "ssl.py", line 215, in recv
File: "ssl.py", line 136, in read

Looking at dump it looks like socket recv got stuck. After digging deep i found that there is no timeout value set on this socket. The class HTTPSConnectionWrapper in SoapAdapter sets timeout value but only if we pass any ssl params. In my case i don't have any extra ssl paramters to pass to pyVim Connect call.
I also looked at connection_pool_timeout but after looking at pyVim Connect call there is provision to set this parameter which is making it set to default 900 secs. I am assuming here even 900 secs is set that is not set on the socket created by httplib hence my WaitForUpdates call stuck. After looking into httplib code if timeout value is not passed then not even global timeout is set.

I also explored WaitForUpdatesEx api. But here I'm not sure if setting maxWaitSeconds would set timeout for the httplib socket.

My suggestion here would be either provide explicit "timeout" parameter to pyVim Connect/SmartConnect api and pass it to HTTPSConnectionWrapper or set some default timeout value to this socket.

Please correct me if I'm missing here something. I would even like to contribute this thing if given a chance.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the socket timeout path from pyVim Connect/SmartConnect into HTTPSConnectionWrapper in pyVmomi/SoapAdapter.py, using the reported WaitForUpdates stack as the entry point. Check whether connection_pool_timeout or WaitForUpdatesEx affects the httplib socket. Done should establish the timeout behavior and define one accepted way to prevent an indefinitely blocked call.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.