canonical / canonical/mlmd-operator

Fix integration test

Open
#22 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
2
Forks
6
Avg merge
1d 9h
Merged PRs (30d)
2

Description

The `test_using_charm` integration test seems to send empty metadata and receive an empty response, giving a false-pass result. This can be observed in [CI runs](https://github.com/canonical/mlmd-operator/runs/7923034478?check_suite_focus=true#step:5:480) and locally:

```
$ tox -e integration -- --model=mlmd-test --keep-models
integration installed: asttokens==2.0.8,attrs==22.1.0,backcall==0.2.0,bcrypt==4.0.0,cachetools==5.2.0,certifi==2022.6.15,cffi==1.15.1,chardet==3.0.4,cryptography==37.0.4,decorator==5.1.1,executing==1.0.0,google-auth==2.11.0,idna==2.10,iniconfig==1.1.1,ipdb==0.13.9,ipython==8.4.0,jedi==0.18.1,Jinja2==3.1.2,jsonschema==3.2.0,juju==3.0.1,jujubundlelib==0.5.7,kubernetes==24.2.0,macaroonbakery==1.3.1,MarkupSafe==2.1.1,matplotlib-inline==0.1.6,mypy-extensions==0.4.3,oauthlib==3.2.0,oci-image==1.0.0,ops==1.2.0,packaging==21.3,paramiko==2.11.0,parso==0.8.3,pexpect==4.8.0,pickleshare==0.7.5,pluggy==1.0.0,prompt-toolkit==3.0.30,protobuf==3.20.1,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pycparser==2.21,Pygments==2.13.0,pymacaroons==0.13.0,PyNaCl==1.5.0,pyparsing==3.0.9,pyRFC3339==1.1,pyrsistent==0.18.1,pytest==7.1.2,pytest-asyncio==0.19.0,pytest-operator==0.22.0,python-dateutil==2.8.2,pytz==2022.2.1,PyYAML==6.0,requests==2.25.0,requests-oauthlib==1.3.1,rsa==4.9,serialized-data-interface==0.3.5,six==1.16.0,stack-data==0.5.0,theblues==0.5.2,toml==0.10.2,tomli==2.0.1,toposort==1.7,traitlets==5.3.0,typing-extensions==4.3.0,typing-inspect==0.8.0,urllib3==1.26.12,wcwidth==0.2.5,websocket-client==1.4.0,websockets==7.0
integration run-test-pre: PYTHONHASHSEED='3683306377'
integration run-test: commands[0] | pytest -vv --tb native --show-capture=no --log-cli-level=INFO -s --model=mlmd-test --keep-models /home/ubuntu/mlmd-operator/tests/integration
============================================================================ test session starts ============================================================================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0 -- /home/ubuntu/mlmd-operator/.tox/integration/bin/python
cachedir: .tox/integration/.pytest_cache
rootdir: /home/ubuntu/mlmd-operator
plugins: asyncio-0.19.0, operator-0.22.0
asyncio: mode=strict
collected 2 items

tests/integration/test_charm.py::test_build_and_deploy SKIPPED (unconditional skip)
tests/integration/test_charm.py::test_using_charm
------------------------------------------------------------------------------ live log setup -------------------------------------------------------------------------------
INFO pytest_operator.plugin:plugin.py:653 Connecting to existing model my-controller:mlmd-test on unspecified cloud
------------------------------------------------------------------------------- live log call -------------------------------------------------------------------------------
INFO root:test_charm.py:39 Using temporary directory /home/ubuntu/mlmd-operator/.tox/integration/tmp/pytest/test_using_charm0
INFO root:test_charm.py:40 cwd = /home/ubuntu/mlmd-operator
INFO root:test_charm.py:41 script_abs_path = /home/ubuntu/mlmd-operator/tests/integration/data/interact_with_mlmd.sh
+ MODEL=mlmd-test
+ echo MODEL=mlmd-test
MODEL=mlmd-test
+ wget https://raw.githubusercontent.com/google/ml-metadata/master/ml_metadata/proto/metadata_store.proto
--2022-08-31 09:33:53-- https://raw.githubusercontent.com/google/ml-metadata/master/ml_metadata/proto/metadata_store.proto
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 36972 (36K) [text/plain]
Saving to: ‘metadata_store.proto’

metadata_store.proto 100%[========================================================================================>] 36.11K --.-KB/s in 0s

2022-08-31 09:33:53 (110 MB/s) - ‘metadata_store.proto’ saved [36972/36972]

+ wget https://raw.githubusercontent.com/google/ml-metadata/master/ml_metadata/proto/metadata_store_service.proto
--2022-08-31 09:33:53-- https://raw.githubusercontent.com/google/ml-metadata/master/ml_metadata/proto/metadata_store_service.proto
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 39192 (38K) [text/plain]
Saving to: ‘metadata_store_service.proto’

metadata_store_service.proto 100%[========================================================================================>] 38.27K --.-KB/s in 0s

2022-08-31 09:33:53 (89.3 MB/s) - ‘metadata_store_service.proto’ saved [39192/39192]

+ wget -O- https://github.com/fullstorydev/grpcurl/releases/download/v1.8.0/grpcurl_1.8.0_linux_x86_64.tar.gz
+ tar -xzv
--2022-08-31 09:33:53-- https://github.com/fullstorydev/grpcurl/releases/download/v1.8.0/grpcurl_1.8.0_linux_x86_64.tar.gz
Resolving github.com (github.com)... 140.82.113.4
Connecting to github.com (github.com)|140.82.113.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/111431261/88be8800-4f53-11eb-94fd-6a7ef1143069?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220831%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220831T093354Z&X-Amz-Expires=300&X-Amz-Signature=34b32b5c3850e69f87bbdeb08d6ef42aa04ac13e68afcb2337fe13692efddfc6&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=111431261&response-content-disposition=attachment%3B%20filename%3Dgrpcurl_1.8.0_linux_x86_64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2022-08-31 09:33:54-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/111431261/88be8800-4f53-11eb-94fd-6a7ef1143069?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220831%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220831T093354Z&X-Amz-Expires=300&X-Amz-Signature=34b32b5c3850e69f87bbdeb08d6ef42aa04ac13e68afcb2337fe13692efddfc6&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=111431261&response-content-disposition=attachment%3B%20filename%3Dgrpcurl_1.8.0_linux_x86_64.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.110.133, 185.199.108.133, 185.199.109.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4986341 (4.8M) [application/octet-stream]
Saving to: ‘STDOUT’

- 0%[ ] 0 --.-KB/s LICENSE
grpcurl
- 100%[========================================================================================>] 4.75M --.-KB/s in 0.1s

2022-08-31 09:33:54 (35.2 MB/s) - written to stdout [4986341/4986341]

+ mkdir -p ml_metadata/proto/
+ mv metadata_store.proto ml_metadata/proto/
++ kubectl get services/mlmd -n mlmd-test -oyaml
++ yq e .spec.clusterIP -
+ SERVICE=10.152.183.202
+ ./grpcurl -v --proto=metadata_store_service.proto --plaintext 10.152.183.202:8080 ml_metadata.MetadataStoreService/GetArtifacts

Resolved method descriptor:
// Gets all the artifacts.
rpc GetArtifacts ( .ml_metadata.GetArtifactsRequest ) returns ( .ml_metadata.GetArtifactsResponse );

Request metadata to send:
(empty)

Response headers received:
accept-encoding: identity,gzip
content-type: application/grpc
grpc-accept-encoding: identity,deflate,gzip

Response contents:
{

}

Response trailers received:
(empty)
Sent 0 requests and received 1 response
PASSED
----------------------------------------------------------------------------- live log teardown -----------------------------------------------------------------------------
INFO pytest_operator.plugin:plugin.py:768 Model status:

Model Controller Cloud/Region Version SLA Timestamp
mlmd-test my-controller myk8s/localhost 2.9.33 unsupported 09:33:54Z

App Version Status Scale Charm Channel Rev Address Exposed Message
mlmd res:oci-image@e2cb9ce active 1 mlmd stable 10 10.152.183.114 no

Unit Workload Agent Address Ports Message
mlmd/0* active idle 10.1.17.61 8080/TCP
```

Contributor guide

Open the contributing guide

Research direction

Start with tests/integration/test_charm.py and tests/integration/data/interact_with_mlmd.sh, then run tox -e integration -- --model=mlmd-test --keep-models to reproduce test_using_charm. Inspect the grpcurl request and response handling so the test cannot pass with empty metadata and an empty response. Done means the integration test validates a meaningful result and fails when the service returns the currently observed empty response.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, kubernetes, python
Domain
api, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.