apache / apache/buildstream

Stack traces when running out of disk space

Open
#660 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/660)
In GitLab by [[Gitlab user @tristanvb]](https://gitlab.com/tristanvb) on Sep 18, 2018, 11:28

## Summary

[//]: # (Summarize the bug encountered concisely)

This is a follow up of issue #623 which is mostly fixed and will not realistically happen now that it is fixed by !775 and !779

## Steps to reproduce

[//]: # (How one can reproduce the issue - this is very important)

To reproduce this, you need to have an artifact on a volume which will run out of space.

You need the artifact being committed to be very large, larger than the headroom afforded by the cache quota algorithm.

When building an element which will exceed the headroom, or if for any reason the same volume has been filled by an external process to BuildStream, at commit time we will get an out of disk exception.

## What is the current bug behavior?

[//]: # (What actually happens)

We get a series of unhandled exceptions printed as BUG messages, as shown in #623

## What is the expected correct behavior?

[//]: # (What you should see instead)

We should get an error message indicating that there was no space left on the device, and we should force terminate any ongoing tasks as a result and exit gracefully.

If `BuildStream` was started with `bst --log-file build.log`, then we should also not see any exceptions due to not being able to add any logs to that master log.

Any ongoing tasks which might be logging to individual log files, should also not trigger exceptions in this case.

## Relevant logs and/or screenshots

[//]: # (Paste any relevant logs - please use code blocks ``` to format console output, logs, and code as it's hard to read otherwise. You can also add a Snippet and link it here. Check the markdown giude on Gitlab for further tips)

See #623

## Possible fixes

[//]: # (If you can, link to the line of code that might be responsible for the problem)

## Other relevant information

[//]: # (Add the BuildStream version. Substitute below the "x" by "1", "2" or the right version. Add more than one version if necessary)

* BuildStream version affected: /milestone %BuildStream_v1.x

----

Contributor guide

Open the contributing guide

Research direction

Start by reviewing issue #623 and reproducing the failure with a large artifact on a volume that has no remaining space, including `bst --log-file build.log`. Done means a clear no-space error, graceful termination of ongoing tasks, and no secondary exceptions from the master or task log files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.