Azure / Azure/azure-cli-extensions

CI doesn't test final built extension packages

Open
#9,425 3 comments 0 reactions 0 assignees View on GitHub
bug Service Attention
Dominant language
Python
Stars
454
Forks
1.7k
Avg merge
2d 19h
Merged PRs (30d)
64

Description

### Describe the bug

This is more a lack of test coverage than a bug in the final product. But the fact CI only tests the local checkout of the code using azdev instead of installing a built package of the extension means we are actually performing zero testing on the code we ship to customers.

You can really easily expose this issue by making an extension rely on files which are not included in the built python wheel. For example, I found this issue by adding code in a new directory without the `__init__.py` file required to pickup that directory in building the package, CI passes with full coverage but the final released package fails with any invocation even `--help`.

Implementing this coverage may be difficult as currently use `azdev test` which as far as I know by design pulls in all of the extension source including things which may not be included in releases, meaning tests might rely on unreleased code/data.

### Related command

Test coverage issue so irrelevant

### Errors

Test coverage issue so irrelevant

### Issue script & Debug output

Test coverage issue so irrelevant

### Expected behavior

Behaviour tested in CI should represent the user experience using the extension with regular az cli not azdev

### Environment Summary

Test coverage issue so irrelevant

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.