google / google/mtail

NFS: Stale file handle - after log cycling occurs.

Open
#505 21 comments 0 reactions 0 assignees View on GitHub
bug help wanted mtail-Log Tailing
Dominant language
Go
Stars
4k
Forks
392
PR merge metrics
No merged PRs in 30d

Description

I'm running mtail as a separate Docker container, mounting an NFS (EFS) volume to which another application writes out it's log files. I noticed however that at some point the mtail containers went to 100% CPU, closer investigation shows that the mtail logs are rapidly filling up with:

```
I0422 19:46:12.453819 13 filestream.go:116] read /logs/app.log: stale NFS file handle
I0422 19:46:12.453841 13 filestream.go:116] read /logs/app.log: stale NFS file handle
I0422 19:46:12.453872 13 filestream.go:116] read /logs/app.log: stale NFS file handle
...
...
```

That explains the 100% CPU at least, I guess. However, I found out that I can trigger this behaviour by simply redeploying the application that is writing out the logs (app.log will be renamed to app.log.1 on deployment and a fresh app.log is written out).

I'm understanding that mtail has some heuristics to attempt to follow files after rotation, is there a particular reason why this is failing when the files are served from NFS?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.