apache / apache/buildstream

Hanging tests due to exceptions in artifact cache

Open
#811 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/811)
In GitLab by [[Gitlab user @jmacarthur]](https://gitlab.com/jmacarthur) on Dec 7, 2018, 10:18

## Summary

Likely only to affect developers but very annoying. Exceptions in the Artifact cache cause the BuildStream tests to lock up. They cannot be interrupted with `^C` and need a `kill -9` in another terminal to kill them.

## Steps to reproduce

Simulate a missing file in `CASCache._get_subdir`, like this:

```
diff --git a/buildstream/_artifactcache/cascache.py b/buildstream/_artifactcache/cascache.py
index 9ca757d4..29cb84a0 100644
--- a/buildstream/_artifactcache/cascache.py
+++ b/buildstream/_artifactcache/cascache.py
[[Gitlab user @]](https://gitlab.com/)[[Gitlab user @]](https://gitlab.com/) -797,6 +797,7 [[Gitlab user @]](https://gitlab.com/)[[Gitlab user @]](https://gitlab.com/) class CASCache():

def _get_subdir(self, tree, subdir):
head, name = os.path.split(subdir)
+ raise CASError("Subdirectory {} not found".format(name))
if head:
tree = self._get_subdir(tree, head)
```

(This exception is raised at the end of this function, and can genuinely be raised if an badly-constructed artifact was placed in the cache.)

Now run the test:

`/setup.py test --addopts "tests/artifactcache/pull.py::test_pull --integration -s"`

This will halt at 'pull'.

## What is the current bug behavior?

Tests lock up and can only be cleared with SIGKILL.

## What is the expected correct behavior?

Details of the exception being raised are visible to the tester.

## Possible fixes

While we can't simply remove it, `ExitStack` in `tests/testutils/runcli.py` is likely to be relevant. In my case, I tracked down the underlying exception by removing ExitStack from `Cli.run` temporarily. This allows the exception text and backtrace to appear on the test output.

* BuildStream version affected: /milestone %BuildStream_v1.3

----

Contributor guide

Open the contributing guide

Research direction

Start with tests/testutils/runcli.py and its ExitStack handling, then inspect buildstream/_artifactcache/cascache.py and reproduce the problem with /setup.py test --addopts "tests/artifactcache/pull.py::test_pull --integration -s". Done means the test no longer hangs when an artifact-cache exception occurs and the exception details and backtrace are visible in the test output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.