apache / apache/buildstream

Crash due to AttributeError when using remote execution

Open
#1,315 3 comments 0 reactions 0 assignees View on GitHub
bug needinfo remote execution
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/1315)
In GitLab by [[Gitlab user @scott]](https://gitlab.com/scott).clarke on May 20, 2020, 13:41

## Summary

[//]: # (Summarize the bug encountered concisely)
I experienced a build failure due to and `AttributeError`, shown below, when using remote execution and having an artifact server defined.

## Steps to reproduce

[//]: # (How one can reproduce the issue - this is very important)
I saw the failure when doing a remote execution build of [this element](https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/blob/valentindavid/bst2/elements/bootstrap-import.bst) with my `project.conf` contaning the lines shown below. I only saw this error when the artifact cache was defined, if commented out this didn't occur.

## What is the current bug behavior?

[//]: # (What actually happens)
Buildstream crashes with an `AttributeError` as seen below.

## What is the expected correct behavior?

[//]: # (What you should see instead)
Buildstream shouldn't crash.

## Relevant logs and/or screenshots

[//]: # (Paste any relevant logs - please use code blocks ``` to format console output, logs, and code as it's hard to read otherwise. You can also add a Snippet and link it here. Check the markdown giude on Gitlab for further tips)
```
An unhandled exception occured:

Traceback (most recent call last):
File "/usr/local/lib64/python3.7/site-packages/buildstream/_scheduler/jobs/job.py", line 709, in child_action
result = self.child_process() # pylint: disable=assignment-from-no-return
File "/usr/local/lib64/python3.7/site-packages/buildstream/_scheduler/jobs/elementjob.py", line 103, in child_process
return self._action_cb(self._element)
File "/usr/local/lib64/python3.7/site-packages/buildstream/_scheduler/queues/buildqueue.py", line 93, in _assemble_element
return element._assemble()
File "/usr/local/lib64/python3.7/site-packages/buildstream/element.py", line 1572, in _assemble
collect = self.assemble(sandbox) # pylint: disable=assignment-from-no-return
File "/usr/lib64/python3.7/contextlib.py", line 119, in __exit__
next(self.gen)
File "/usr/local/lib64/python3.7/site-packages/buildstream/sandbox/sandbox.py", line 328, in batch
batch.execute()
File "/usr/local/lib64/python3.7/site-packages/buildstream/sandbox/_sandboxreapi.py", line 236, in execute
if self.sandbox.run(["sh", "-c", "-e", self.script], self.flags, cwd=first.cwd, env=first.env) != 0:
File "/usr/local/lib64/python3.7/site-packages/buildstream/sandbox/sandbox.py", line 282, in run
return self._run(command, flags, cwd=cwd, env=env)
File "/usr/local/lib64/python3.7/site-packages/buildstream/sandbox/_sandboxreapi.py", line 106, in _run
action_result = self._execute_action(action, flags) # pylint: disable=assignment-from-no-return
File "/usr/local/lib64/python3.7/site-packages/buildstream/sandbox/_sandboxremote.py", line 330, in _execute_action
artifactcache.fetch_missing_blobs(project, local_missing_blobs)
File "/usr/local/lib64/python3.7/site-packages/buildstream/_artifactcache.py", line 408, in fetch_missing_blobs
missing_blobs = self.cas.fetch_blobs(remote, missing_blobs, allow_partial=True)
File "/usr/local/lib64/python3.7/site-packages/buildstream/_cas/cascache.py", line 670, in fetch_blobs
batch.add(digest)
File "/usr/local/lib64/python3.7/site-packages/buildstream/_cas/casremote.py", line 103, in add
self._request.instance_name = self._remote.local_cas_instance_name
AttributeError: 'ArtifactRemote' object has no attribute 'local_cas_instance_name'
```
```
artifacts:
url: https://public.aws.celduin.co.uk

remote-execution:
execution-service:
url: https://push.cache.com:30443
instance-name: remote-execution
server-cert: server.crt
client-cert: push.crt
client-key: push.key
storage-service:
url: https://push.cache.com:30443
instance-name: remote-execution
server-cert: server.crt
client-cert: push.crt
client-key: push.key
action-cache-service:
url: https://push.cache.com:30443
instance-name: remote-execution
server-cert: server.crt
client-cert: push.crt
client-key: push.key
```

## Possible fixes

[//]: # (If you can, link to the line of code that might be responsible for the problem)

## Other relevant information

[//]: # (Add the BuildStream version. Substitute below the "x" by "1", "2" or the right version. Add more than one version if necessary)

* BuildStream version affected:
This happened in a container built from the nightly docker image, the latest buildstream commit at the time of building was: 99063c732b56fb92f6c8b8c32bdb66eb4fb015e6
```
/src> bst --version
1.93.3+76.g99063c73
```

----

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the shown project.conf remote-execution and artifacts settings, then trace the stack from sandbox/_sandboxremote.py through _artifactcache.py, _cas/cascache.py, and _cas/casremote.py. Compare the remote artifact-cache objects involved and verify that a remote-execution build with an artifact server no longer crashes with AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.