Azure / Azure/azure-cli

TestSDK utilities.py fails to convert byte to string for test recording

Open
#10,705 9 comments 0 reactions 1 assignee Claimed by @qwordy View on GitHub
feature-request Test Framework
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

> ### `az feedback` auto-generates most of the information requested below, as of CLI version
2.0.73

**Describe the bug**
When implementing an E2E test scenario to upload a zip file to a storage account (using **StorageAccountPreparer**), the _py3_byte_to_str() function inside testsdk/utilities.py fails to convert byte to string.

This affects the recording in `env\lib\site-packages\azure_devtools\scenario_tests\base.py:157` if the test is decorated by StorageAccountPreparer and the test-covered code contains request to upload zip file.

**To Reproduce**
1. Paste the following e2e test scenario in `azure-cli\src\azure-cli\azure\cli\command_modules\appservice\tests\latest\test_webapp_commands.py`
```python
class FunctionappRemoteBuildScenarioTest(ScenarioTest):
@ResourceGroupPreparer(name_prefix='azurecli-functionapp-c-e2e-remotebuild')
@StorageAccountPreparer(sku='Standard_LRS')
def test_functionapp_remote_build(self, resource_group, storage_account):
functionapp_name = self.create_random_name('functionapp-remotebuild-test', 40)
plan_name = self.create_random_name('functionapp-remotebuild-plan', 40)
zip_file = os.path.join(TEST_DIR, 'test.zip')
self.cmd('functionapp plan create -g {} -n {} --sku S1 --is-linux true'.format(resource_group, plan_name))
self.cmd('functionapp create -g {} -n {} --plan {} -s {} --os-type Linux --runtime python'.format(resource_group, functionapp_name, plan_name, storage_account))
self.cmd('functionapp deployment source config-zip -g {} -n {} --src "{}"'.format(resource_group, functionapp_name, zip_file)).assert_with_checks([
JMESPathCheck('status', 4),
JMESPathCheck('deployer', 'Push-Deployer'),
JMESPathCheck('message', 'Created via a push deployment'),
JMESPathCheck('complete', True)
])
```
2. Activate your virtual environment, setup the development environment.
3. Run `azdev test test_functionapp_remote_build`
4. The e2e test fails with the following traceback (in appendix).

**Expected behavior**
1. The test should pass without warning
2. A new recording entry should be generated.

**Environment summary**
Install Method (e.g. pip, interactive script, apt-get, Docker, MSI, edge build) / CLI version (`az --version`) / OS version / Shell Type (e.g. bash, cmd.exe, Bash on Windows)
- Setup by the azdev

**Appendix (Error Traceback)**
```
(env) C:\Users\hazeng\Projects\GitHub\azure-cli [hazeng-pr4-remotebuild +3 ~4 -1 !]> azdev test test_functionapp_remote_build

=============
| Run Tests |
=============

The tests are set to run against current profile latest.

test index found: C:\Users\hazeng\.azdev\env_config\Users\hazeng\Projects\Github\azure-cli\env\test_index\latest.json

TESTS: test_functionapp_remote_build

============================================================================================================================ test session starts =============================================================================================================================
platform win32 -- Python 3.6.8, pytest-4.4.2, py-1.8.0, pluggy-0.13.0
rootdir: c:\users\hazeng\projects\github\azure-cli\src\azure-cli
plugins: forked-1.0.2, xdist-1.29.0
gw0 [1] / gw1 [1] / gw2 [1] / gw3 [1] / gw4 [1] / gw5 [1] / gw6 [1] / gw7 [1]
F [100%]
================================================================================================================================== FAILURES ==================================================================================================================================
______________________________________________________________________________________________________ FunctionappRemoteBuildScenarioTest.test_functionapp_remote_build ______________________________________________________________________________________________________
[gw0] win32 -- Python 3.6.8 C:\Users\hazeng\Projects\Github\azure-cli\env\Scripts\python.exe

self = , resource_group = 'azurecli-functionapp-c-e2e-remotebuildqryg4sju63qtegcbb3z7nxxj67s376644cjln'
storage_account = 'clitestwditlbpoox6lbg4dh'

@ResourceGroupPreparer(name_prefix='azurecli-functionapp-c-e2e-remotebuild', location='westus')
@StorageAccountPreparer(sku='Standard_LRS')
def test_functionapp_remote_build(self, resource_group, storage_account):
plan_name = self.create_random_name('functionapp-remotebuild-plan', 40)
functionapp_name = self.create_random_name('functionappconsumption', 40)
zip_file = os.path.join(TEST_DIR, 'test.zip')
self.cmd('functionapp plan create -g {} -n {} --sku S1 --is-linux true'.format(resource_group, plan_name))
self.cmd('functionapp create -g {} -n {} -p {} -s {} --os-type Linux --runtime python'
.format(resource_group, functionapp_name, plan_name, storage_account))
self.cmd('webapp deployment source config-zip -g {} -n {} --src "{}"'
> .format(resource_group, functionapp_name, zip_file))

src\azure-cli\azure\cli\command_modules\appservice\tests\latest\test_webapp_commands.py:1431:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src\azure-cli-testsdk\azure\cli\testsdk\base.py:161: in cmd
return execute(self.cli_ctx, command, expect_failure=expect_failure).assert_with_checks(checks)
src\azure-cli-testsdk\azure\cli\testsdk\base.py:201: in __init__
self._in_process_execute(cli_ctx, command, expect_failure=expect_failure)
src\azure-cli-testsdk\azure\cli\testsdk\base.py:264: in _in_process_execute
raise ex.exception
env\lib\site-packages\knack\cli.py:206: in invoke
cmd_result = self.invocation.execute(args)
src\azure-cli-core\azure\cli\core\commands\__init__.py:603: in execute
raise ex
src\azure-cli-core\azure\cli\core\commands\__init__.py:661: in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
src\azure-cli-core\azure\cli\core\commands\__init__.py:654: in _run_job
env\lib\site-packages\six.py:693: in reraise
raise value
src\azure-cli-core\azure\cli\core\commands\__init__.py:631: in _run_job
result = cmd_copy(params)
src\azure-cli-core\azure\cli\core\commands\__init__.py:306: in __call__
return self.handler(*args, **kwargs)
src\azure-cli-core\azure\cli\core\__init__.py:485: in default_command_handler
return op(**command_args)
src\azure-cli\azure\cli\command_modules\appservice\custom.py:364: in enable_zip_deploy
res = requests.post(zip_url, data=zip_content, headers=headers, verify=not should_disable_connection_verify())
env\lib\site-packages\requests\api.py:116: in post
return request('post', url, data=data, json=json, **kwargs)
env\lib\site-packages\requests\api.py:60: in request
return session.request(method=method, url=url, **kwargs)
env\lib\site-packages\requests\sessions.py:533: in request
resp = self.send(prep, **send_kwargs)
env\lib\site-packages\requests\sessions.py:646: in send
r = adapter.send(request, **kwargs)
env\lib\site-packages\requests\adapters.py:449: in send
timeout=timeout
env\lib\site-packages\urllib3\connectionpool.py:600: in urlopen
chunked=chunked)
env\lib\site-packages\urllib3\connectionpool.py:377: in _make_request
httplib_response = conn.getresponse(buffering=True)
env\lib\site-packages\vcr\stubs\__init__.py:220: in getresponse
if self.cassette.can_play_response_for(self._vcr_request):
env\lib\site-packages\vcr\cassette.py:254: in can_play_response_for
request = self._before_record_request(request)
env\lib\site-packages\vcr\config.py:253: in before_record_request
request = function(request)
env\lib\site-packages\azure_devtools\scenario_tests\base.py:157: in _process_request_recording
request = processor.process_request(request)
src\azure-cli-testsdk\azure\cli\testsdk\utilities.py:85: in process_request
body_string = _py3_byte_to_str(request.body)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

byte_or_str = b'PK\x03\x04\x14\x00\x00\x00\x08\x00\x04dQK\xc1\x98\x14\x97S\x00\x00\x00p\x00\x00\x00\t\x00\x00\x00test.html\xb3Qt\xf1...\x00 \x00\x00\x00\x00\x00\x00\x00test.htmlPK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x007\x00\x00\x00z\x00\x00\x00\x00\x00'

def _py3_byte_to_str(byte_or_str):
import logging
logger = logging.getLogger()
logger.warning(type(byte_or_str))
try:
> return str(byte_or_str, 'utf-8') if isinstance(byte_or_str, bytes) else byte_or_str
E UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc1 in position 14: invalid start byte

src\azure-cli-testsdk\azure\cli\testsdk\utilities.py:57: UnicodeDecodeError
---------------------------------------------------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------------------------------------------------
WARNING: The default kind for created storage account will change to 'StorageV2' from 'Storage' in future
WARNING: Application Insights "functionappconsumptionnqp6srbglvy5vq5eat" was created for this Function App. You can visit https://portal.azure.com/#resource/subscriptions/fdaf2e05-dba9-4a35-8130-41d1a855a739/resourceGroups/azurecli-functionapp-c-e2e-remotebuildqryg4sju63qtegcbb3z7nxxj67s376644cjln/providers/microsoft.insights/components/functionappconsumptionnqp6srbglvy5vq5eat/overview to view your Application Insights component
WARNING: Getting scm site credentials for zip deployment
WARNING: Starting zip deployment. This operation can take a while to complete ...
----------------------------------------------------------------------------- generated xml file: C:\Users\hazeng\.azdev\env_config\Users\hazeng\Projects\Github\azure-cli\env\test_results.xml ------------------------------------------------------------------------------
========================================================================================================================= 1 failed in 72.77 seconds ==========================================================================================================================

Results
=========

Time: 72.699 sec Tests: 1 Skipped: None Failures: 1 Errors: 0

FAILURES
==========

test_webapp_commands.FunctionappRemoteBuildScenarioTest.test_functionapp_remote_build
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.