WaitForUpdates stuck for more than 2 days
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 763
- PR merge metrics
- No merged PRs in 30d
Description
[pyvmomi version = 6.0.0.2016.6]
I'm facing one issue while creating snapshot. The api got stuck for more than 2 days or in some cases until i kill the process. I did some analysis here and need your opinion on that. Let's start with thread dump :
File: "pyVmomi/VmomiSupport.py", line 560, in call
File: "pyVmomi/VmomiSupport.py", line 383, in _InvokeAccessor
File: "pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor
File: "pyVmomi/VmomiSupport.py", line 566, in
File: "pyVmomi/VmomiSupport.py", line 375, in _InvokeMethod
File: "pyVmomi/SoapAdapter.py", line 1282, in InvokeMethod
File: "pyVmomi/SoapAdapter.py", line 783, in Deserialize
File: "pyVmomi/SoapAdapter.py", line 456, in ReadDocument
File: "httplib.py", line 522, in read
File: "httplib.py", line 569, in _read_chunked
File: "httplib.py", line 619, in _safe_read
File: "socket.py", line 377, in read
File: "ssl.py", line 215, in recv
File: "ssl.py", line 136, in read
Looking at dump it looks like socket recv got stuck. After digging deep i found that there is no timeout value set on this socket. The class HTTPSConnectionWrapper in SoapAdapter sets timeout value but only if we pass any ssl params. In my case i don't have any extra ssl paramters to pass to pyVim Connect call.
I also looked at connection_pool_timeout but after looking at pyVim Connect call there is provision to set this parameter which is making it set to default 900 secs. I am assuming here even 900 secs is set that is not set on the socket created by httplib hence my WaitForUpdates call stuck. After looking into httplib code if timeout value is not passed then not even global timeout is set.
I also explored WaitForUpdatesEx api. But here I'm not sure if setting maxWaitSeconds would set timeout for the httplib socket.
My suggestion here would be either provide explicit "timeout" parameter to pyVim Connect/SmartConnect api and pass it to HTTPSConnectionWrapper or set some default timeout value to this socket.
Please correct me if I'm missing here something. I would even like to contribute this thing if given a chance.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the socket timeout path from pyVim Connect/SmartConnect into HTTPSConnectionWrapper in pyVmomi/SoapAdapter.py, using the reported WaitForUpdates stack as the entry point. Check whether connection_pool_timeout or WaitForUpdatesEx affects the httplib socket. Done should establish the timeout behavior and define one accepted way to prevent an indefinitely blocked call.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100