apache / apache/buildstream

Artifact related bugs reported as user facing errors

Open
#1,016 1 comment 0 reactions 0 assignees View on GitHub
bug logging
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/1016)
In GitLab by [[Gitlab user @tristanvb]](https://gitlab.com/tristanvb) on May 6, 2019, 06:03

While building, I am getting errors as such reported:

```
[00:00:00][????????][build:sdk/glib.bst ] FAILURE Attempt to access unavailable artifact: [Errno 2] No such file or directory: '/home/tristan/.cache/buildstream/artifacts/cas/refs/heads/gnome/sdk-glib/d371839c675f4be8c70f9f212093531699e77baf6500e8e56d8dbd7270607068'
```

A missing artifact is actually a bug, not an error; the user cannot fix this, and it is not a system error, it is clearly our fault if this error is ever seen.

Currently we have a contorted story around these errors, and we mitigate this by calling `Element.__assert_cached()` in some places, but not all places.

I think this is backwards and unsafe, we should reverse this such that the underlying `CAS` errors do not inherit from `BstError`, and have `ArtifactCache` only handle the recoverable errors and turn them into a `BstError` deriving `ArtifactError`.

## In bst-1.2

When staging a built workspace with a missing artifact (after replacing a `raise ArtifactError` with a `raise AssertionError`), we get the following codepath which leads to a bug misreported as an error:

```
An unhandled exception occured:

Traceback (most recent call last):
File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 462, in resolve_ref
with open(refpath, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/tristan/.cache/buildstream/artifacts/cas/refs/heads/gnome/core-gdm/72d9b4e1bcfe9742ea52c7d5f76b62886627a90534a6ffc908b609badc650f3c'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 106, in extract
tree = self.resolve_ref(ref, update_mtime=True)
File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 471, in resolve_ref
raise ArtifactError("Attempt to access unavailable artifact: {}".format(e)) from e
buildstream._exceptions.ArtifactError: Attempt to access unavailable artifact: [Errno 2] No such file or directory: '/home/tristan/.cache/buildstream/artifacts/cas/refs/heads/gnome/core-gdm/72d9b4e1bcfe9742ea52c7d5f76b62886627a90534a6ffc908b609badc650f3c'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/job.py", line 413, in _child_action
result = self.child_process()
File "/codethink/GNOME/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 94, in child_process
return self._action_cb(self._element)
File "/codethink/GNOME/buildstream/buildstream/_scheduler/queues/buildqueue.py", line 35, in process
return element._assemble()
File "/codethink/GNOME/buildstream/buildstream/element.py", line 1523, in _assemble
self.stage(sandbox)
File "/codethink/GNOME/buildstream/buildstream/buildelement.py", line 165, in stage
self.stage_dependency_artifacts(sandbox, Scope.BUILD)
File "/codethink/GNOME/buildstream/buildstream/element.py", line 704, in stage_dependency_artifacts
old_dep_keys = self.__get_artifact_metadata_dependencies(workspace.last_successful)
File "/codethink/GNOME/buildstream/buildstream/element.py", line 2442, in __get_artifact_metadata_dependencies
artifact_base, key = self.__extract(key)
File "/codethink/GNOME/buildstream/buildstream/element.py", line 2394, in __extract
return (self.__artifacts.extract(self, key), key)
File "/codethink/GNOME/buildstream/buildstream/_artifactcache/cascache.py", line 108, in extract
raise AssertionError(str(e)) from e
AssertionError: Attempt to access unavailable artifact: [Errno 2] No such file or directory: '/home/tristan/.cache/buildstream/artifacts/cas/refs/heads/gnome/core-gdm/72d9b4e1bcfe9742ea52c7d5f76b62886627a90534a6ffc908b609badc650f3c'
```

## In master

Here I have not checked the stack trace yet, but I have verified that the code still runs in the same fashion: We raise a `CASCacheError` which derives from `BstError`, and we sprinkle `self.__assert_cached()` statements around `element.py` in the hopes of covering any case where we're about to access an artifact which doesnt exist, instead of having the underlying `CASCacheError` be treated as a simple exception, and reporting the recoverable errors as `ArtifactErrors`.

In this codepath, we would hit the following:
* `Element.stage_dependency_artifacts()`
* This instantiate an `Artifact` for the old (missing) workspaced artifact
* This will call into `Artifact.get_metadata_dependencies()`
* This will end up calling `Artifact.get_artifact_directory()`
* Here we explicitly raise an `ArtifactError`, which will inform the user that the artifact is not available, instead of reporting a bug with a stack trace.

Instead, we should have a `CASCacheError` which is *not* a `BstError`, and we should simply not `except CASCacheError` in that case, ensuring that the missing artifact is reported as a `BUG`.

Contributor guide

Open the contributing guide

Research direction

Start with buildstream/_artifactcache/cascache.py and buildstream/element.py, tracing Element.stage_dependency_artifacts() through Artifact.get_metadata_dependencies() and Artifact.get_artifact_directory(). Review the exception handling and existing __assert_cached() calls, then reproduce the missing-artifact path. Done means missing artifacts are reported as bugs while recoverable ArtifactCache errors remain user-facing errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.