lbryio / lbryio/lbry-sdk

Stopping lbrynet fails on full disk

Open
#1,738 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.2k
Forks
493
PR merge metrics
No merged PRs in 30d

Description

## The Issue

Trying to stop the daemon when the disk is full fails.
Trying to stop it using systemd also leaves a single process running which can only be solved with kill -9
Lbrynet seems to be using disk space at runtime?

### Steps to reproduce
1. fill the disk
2. run brynet stop
3. when it fails, check disk space
4. forcefully terminate the process
5. watch disk space again

### Expected behaviour
The daemon stops

### Actual behaviour
./lbrynet stop
[15609] INTERNAL ERROR: cannot create temporary directory!

```
$ sudo service lbrynet stop
$ ps x
PID TTY STAT TIME COMMAND
...
14500 ? Sl 45:37 /opt/lbry/lbrynet start
15964 pts/0 R+ 0:00 ps x
```

## System Configuration
```
{
"id": null,
"jsonrpc": "2.0",
"result": {
"build": "release",
"desktop": "Unknown",
"distro": {
"codename": "Xenial Xerus",
"id": "ubuntu",
"like": "debian",
"version": "16.04",
"version_parts": {
"build_number": "",
"major": "16",
"minor": "04"
}
},
"lbrynet_version": "0.30.3",
"lbryschema_version": "0.0.16",
"os_release": "4.4.0-138-generic",
"os_system": "Linux",
"platform": "Linux-4.4.0-138-generic-x86_64-with-debian-stretch-sid",
"processor": "x86_64",
"python_version": "3.7.1"
}
}
```

## Anything Else
It seems that when I force stop the daemon, in some cases, a lot of space is freed up. Why is that?

```
[lbry@whatever:~] $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 78G 78G 0 100% /

[lbry@whatever:~] $ sudo service lbrynet stop

... out of the 2 lbrynet processes only 1 terminates, so one needs to be force killed
[lbry@whatever:~] $ kill -9 14500

[lbry@whatever:~] $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 78G 49G 29G 64% /

```

## Internal Use

### Acceptance Criteria
1.
2.
3.

### Definition of Done
- [ ] Tested against acceptance criteria
- [ ] Tested against the assumptions of user story
- [ ] The project builds without errors
- [ ] Unit tests are written and passing
- [ ] Tests on devices/browsers listed in the issue have passed
- [ ] QA performed & issues resolved
- [ ] Refactoring completed
- [ ] Any configuration or build changes documented
- [ ] Documentation updated
- [ ] Peer Code Review performed

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure by filling the disk, running `lbrynet stop`, and comparing the `service lbrynet stop` process list and disk usage before and after force-killing the remaining process. Trace the daemon shutdown path from the `lbrynet stop` and systemd/service entry points; done means the daemon stops cleanly on a full disk, no process remains, and regression coverage is added.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
backend, cli, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.