Azure / Azure/azure-cli

Assertion message from JMESPathCheck is misleading when actual result evaluates to False.

Open
#29,697 1 comment 0 reactions 1 assignee Claimed by @jsntcy View on GitHub
Azure CLI Team potential-pruning question
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

### Describe the bug

When running scenario tests for an extension I'm writing, with a check like this:
```
self.cmd(f"az some_command --output json", checks=[
self.check("thing", True)
])
```

The exception when the actual value of `thing` is `False` is misleading:
```
> raise JMESPathCheckAssertionError(self._query, self._expected_result, 'None',
execution_result.output)
E azure.cli.testsdk.exceptions.JMESPathCheckAssertionError: Query 'thing' doesn't yield expected value 'True', instead the actual value is 'None'. Data:
E {
E "otherThing": "value",
E "thing": false
E }
```

It would be better in this case to say "the actual value is `False`".

We encounter similar issues with empty strings, empty lists, empty dictionaries, the number 0 and generally anything that evaluates to `False`.

I've created a [pull request](https://github.com/Azure/azure-cli/pull/29696) with more information about why this happens and a proposed fix.

### Related command

azdev test

### Errors

> raise JMESPathCheckAssertionError(self._query, self._expected_result, 'None',
execution_result.output)
E azure.cli.testsdk.exceptions.JMESPathCheckAssertionError: Query 'thing' doesn't yield expected value 'True', instead the actual value is 'None'. Data:
E {
E "otherThing": "value",
E "thing": false
E }
```

### Issue script & Debug output

I feel this is unnecessary because the error message makes it clear (and see linked pull request).

### Expected behavior

The error message should say the real value of the actual result -- In the provided example, that would be `'False'`.

### Environment Summary

az --version
azure-cli 2.63.0

core 2.63.0
telemetry 1.1.0

Extensions:
aks-preview 7.0.0b3
azure-devops 1.0.1
0.0.1 (dev) /workspace/code///src/

Dependencies:
msal 1.30.0
azure-mgmt-resource 23.1.1

Python location '/workspace/code///.venv/bin/python'
Extensions directory '/home//.azure/cliextensions'
Development extension sources:
/workspace/code//

Python (Linux) 3.9.19 (main, Jul 31 2024, 03:47:41)
[GCC 11.2.0]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.