Graylog2 / Graylog2/graylog2-server

log rotation does not release deleted files / holds disk space

Open
#6,532 1 comment 0 reactions 0 assignees View on GitHub
bug infrastructure to-verify triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior
The logfile rotation should release the file handle and free disk space.

## Current Behavior
When many logs are created within a short time, the filesystem handles and disk space is only release on Graylog restart.

```
[gl2adm@rsv01 graylog]$ ll /var/log/graylog/
-rw-r--r--. 1 gl2adm gl2adm 79650 Sep 18 17:45 graylog-server-1.log.gz
-rw-r--r--. 1 gl2adm gl2adm 83328 Sep 18 17:50 graylog-server-2.log.gz
-rw-r--r--. 1 gl2adm gl2adm 71821 Sep 18 17:51 graylog-server-3.log.gz
-rw-r--r--. 1 gl2adm gl2adm 76595 Sep 18 17:54 graylog-server-4.log.gz
-rw-r--r--. 1 gl2adm gl2adm 77732 Sep 18 17:59 graylog-server-5.log.gz

[gl2adm@rsv01 ~]$ lsof | grep -E 'graylog.*\(deleted\)'
java 15931 gl2adm 1w REG 253,4 1630832143 275859 /var/log/graylog/graylog-server-5.log (deleted)
java 15931 gl2adm 2w REG 253,4 1630832143 275859 /var/log/graylog/graylog-server-5.log (deleted)
java 15931 3969 gl2adm 1w REG 253,4 1630832143 275859 /var/log/graylog/graylog-server-5.log (deleted)
```

this might be limited to xfs filesystems, but not sure on that.

## Steps to Reproduce (for bugs)
1. us this [log4j.xml](https://github.com/Graylog2/graylog2-server/files/3689420/log4j.xml.txt)
2. have Graylog rotate the logs very fast (see timestamps above)

## Context
When you are in a high log situation it is very likely that the reason is high load, what means you can't restart Graylog to free up the disk space. But Graylog will stop working if the logging partition does no free space available.

## Your Environment

* Graylog Version: 3

[z#842704]

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.