Azure / Azure/azure-functions-host
File.Move for Linux dedicated function apps results in lost telemetry
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
Functions logs are written to file by LWAS to guarantee fault tolerance, and the files are periodically rotated so that we do not reach excessive sizes. Our log forwarding service - which is responsible for reading the log files and forwarding the data to our telemetry pipeline - listens for a specific inotify event associated with rotations. When it receives this event, it registers the log file as rotated and searches for the new log file. The File.Move API that is being used to rotate our functions logs is not emitting the correct inotify event, resulting in the log forwarder not being signaled to look for the new logs. By default, our platform initiates periodic restarts of the log forwarding service, which mitigates this issue. However, it can still cause log loss in high volume workloads.
Problem method is [LinuxAppServiceFileLogger.RollFiles()](https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script.WebHost/Diagnostics/LinuxAppServiceFileLogger.cs#L146).
Solution: Use system calls to make sure that the inotify event is emitted.
Contributor guide
Assessment
This issue has not been assessed yet.