Azure / Azure/azure-cli-extensions
CI doesn't test final built extension packages
- Dominant language
- Python
- Stars
- 454
- Forks
- 1.7k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 64
Description
### Describe the bug
This is more a lack of test coverage than a bug in the final product. But the fact CI only tests the local checkout of the code using azdev instead of installing a built package of the extension means we are actually performing zero testing on the code we ship to customers.
You can really easily expose this issue by making an extension rely on files which are not included in the built python wheel. For example, I found this issue by adding code in a new directory without the `__init__.py` file required to pickup that directory in building the package, CI passes with full coverage but the final released package fails with any invocation even `--help`.
Implementing this coverage may be difficult as currently use `azdev test` which as far as I know by design pulls in all of the extension source including things which may not be included in releases, meaning tests might rely on unreleased code/data.
### Related command
Test coverage issue so irrelevant
### Errors
Test coverage issue so irrelevant
### Issue script & Debug output
Test coverage issue so irrelevant
### Expected behavior
Behaviour tested in CI should represent the user experience using the extension with regular az cli not azdev
### Environment Summary
Test coverage issue so irrelevant
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.