google / google/codeworld

Disk space leaking on codeworld server

Open
#882 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Haskell
Stars
1.3k
Forks
201
PR merge metrics
No merged PRs in 30d

Description

The CodeWorld server ran out of disk space tonight. A build was aborted for lack of space to copy a file.

Observations:

1. There were a huge number of files in /tmp from various builds - both full builds and error checks. I believe these come from builds that are aborted and not cleaned up. They aren't huge by themselves, but there were too many to count.
2. Access logs in `codeworld/log` were up to around a gigabyte. We don't ever delete old logs right now. I should probably not bother with access logs anyway, since nginx already keeps them.

There is still a lot there:

cdsmith@codeworld-baker:~/codeworld$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.4G 0 7.4G 0% /dev
tmpfs 1.5G 155M 1.4G 11% /run
/dev/sda1 9.8G 8.5G 832M 92% /
tmpfs 7.4G 0 7.4G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.4G 0 7.4G 0% /sys/fs/cgroup
codeworld-fs-vm:/codeworld-data 961G 11G 951G 2% /var/local/codeworld
cdsmith@codeworld-baker:~/codeworld$ sudo du -h -d 1 --exclude=/var/local/codeworld /
0 /proc
0 /dev
155M /run
4.0K /mnt
5.7G /home
20K /root
8.0M /bin
910M /usr
4.0K /media
4.0K /srv
6.6M /sbin
1.5G /var
0 /sys
34M /boot
225M /lib
4.0K /lib64
16K /lost+found
3.8M /etc
4.0K /opt
96M /tmp
8.6G /

So there's definitely more to clean up.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.