apache / apache/buildstream

Removing artifact cache state dependency in integration tests

Open
#878 1 comment 0 reactions 0 assignees View on GitHub
enhancement tests
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/878)
In GitLab by [[Gitlab user @raoul]](https://gitlab.com/raoul).hidalgocharman on Jan 22, 2019, 11:36

## Background

[//]: # (Provide a background or the root/source that justifies this task or action.)
For integration tests, an artifact cache is shared so that elements don't have to be built again. However this has resulted in tests that are dependent on the state of the artifact cache, meaning reordering or changing tests may cause following tests to fail. There are also tests that pass which don't make complete sense because they rely on the results of a previous test.

For example in the test `test/integration/cachedfail.py::test_push_cached_fail`, for the second parameter where it quits on error it ends up pushing to the artifact cache, because the element is already built, but if you run the second parameter just by itself the test fails. We however do still want to share artifacts for integration tests where possible, as not doing so roughly doubles the time the test suite takes to run.

## Task description

[//]: # (Short summary of the action to be executed)

* [ ] The `cli_integration` context manager should be expanded to allow tests to specify artifacts we want removed before starting a test.

## Acceptance Criteria

[//]: # (Acceptance criteria should follow the S.M.A.R.T. principle https://en.wikipedia.org/wiki/SMART_criteria )
Integration tests should not be dependent on the state of the artifact cache

----

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the cli_integration context manager and test/integration/cachedfail.py::test_push_cached_fail. Trace how the shared artifact cache affects parameterized integration tests, then verify that tests can remove specified artifacts before running without making the suite rebuild everything. Done means integration tests no longer depend on artifact-cache state while shared artifacts remain usable where appropriate.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.