Azure / Azure/azure-cli

Azure CLI connection timeout when running a long test using the ScenarioTest framework

Open
#26,983 6 comments 0 reactions 2 assignees Claimed by @wangzelin007 View on GitHub
act-identity-squad Auto-Assign Auto-Resolve bug Graph
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

### Describe the bug

My team is in the process of creating a new Azure CLI extension. As part of creating this extension, we wanted to add integration tests which would test our new commands. According to this document: https://github.com/Azure/azure-cli/blob/dev/doc/authoring_tests.md in the azure-cli repository, I should be able to author integration tests by taking advantage of the ScenarioTest class.

When I run `azdev test aosm` the publish command in our extension manages to deploy a few resources but fails after a few minutes with the error attached below.

Interestingly, if I reduce the number of deployments that need to be done by this command, the command manages to reach completion. This (and the error) suggest to me that the testing framework implements some sort of timeout which I cannot modify and which does not allow my command to complete running and succeed. Is this correct? Is there anything I can do to remove this timeout condition?

### Related command

`azdev test aosm`

### Errors

```
self =
request =
kwargs = {'stream': False}, response = None
error = ServiceResponseError("('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))")
connection_timeout = 300, timeout = (300, 300), read_timeout = 300

def send(self, request, **kwargs): # type: ignore
"""Send request object according to configuration.

:param request: The request object to be sent.
:type request: ~azure.core.pipeline.transport.HttpRequest
:return: An HTTPResponse object.
:rtype: ~azure.core.pipeline.transport.HttpResponse

:keyword requests.Session session: will override the driver session and use yours.
Should NOT be done unless really required. Anything else is sent straight to requests.
:keyword dict proxies: will define the proxy to use. Proxy is a dict (protocol, url)
"""
self.open()
response = None
error = None # type: Optional[AzureErrorUnion]

try:
connection_timeout = kwargs.pop('connection_timeout', self.connection_config.timeout)

if isinstance(connection_timeout, tuple):
if 'read_timeout' in kwargs:
raise ValueError('Cannot set tuple connection_timeout and read_timeout together')
_LOGGER.warning("Tuple timeout setting is deprecated")
timeout = connection_timeout
else:
read_timeout = kwargs.pop('read_timeout', self.connection_config.read_timeout)
timeout = (connection_timeout, read_timeout)
response = self.session.request( # type: ignore
request.method,
request.url,
headers=request.headers,
data=request.data,
files=request.files,
verify=kwargs.pop('connection_verify', self.connection_config.verify),
timeout=timeout,
cert=kwargs.pop('connection_cert', self.connection_config.cert),
allow_redirects=False,
**kwargs)
response.raw.enforce_content_length = True

except (urllib3.exceptions.NewConnectionError, urllib3.exceptions.ConnectTimeoutError) as err:
error = ServiceRequestError(err, error=err)
except requests.exceptions.ReadTimeout as err:
error = ServiceResponseError(err, error=err)
except requests.exceptions.ConnectionError as err:
if err.args and isinstance(err.args[0], urllib3.exceptions.ProtocolError):
error = ServiceResponseError(err, error=err)
else:
error = ServiceRequestError(err, error=err)
except requests.exceptions.ChunkedEncodingError as err:
msg = err.__str__()
if 'IncompleteRead' in msg:
_LOGGER.warning("Incomplete download: %s", err)
error = IncompleteReadError(err, error=err)
else:
_LOGGER.warning("Unable to stream download: %s", err)
error = HttpResponseError(err, error=err)
except requests.RequestException as err:
error = ServiceRequestError(err, error=err)

if error:
> raise error
E azure.core.exceptions.ServiceResponseError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

../../../../.virtualenvs/az-cli-env/lib/python3.8/site-packages/azure/core/pipeline/transport/_requests_basic.py:361: ServiceResponseError
```

### Issue script & Debug output

no different than the above error message

### Expected behavior

I expect the test to pass.

### Environment Summary

azure-cli 2.48.1 *

core 2.48.1 *
telemetry 1.0.8

Extensions:
connectedk8s 1.3.5
customlocation 0.1.3
k8s-extension 1.3.6
resource-graph 2.1.0
aosm 0.1.0 (dev) /home/developer/Projects/azure-cli-extensions/src/aosm

Dependencies:
msal 1.20.0
azure-mgmt-resource 22.0.0

Python location '/home/developer/.virtualenvs/az-cli-env/bin/python'
Extensions directory '/home/developer/.azure/cliextensions'
Development extension sources:
/home/developer/Projects/azure-cli-extensions

Python (Linux) 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0]

Legal docs and information: aka.ms/AzureCliLegal

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.