Azure / Azure/azure-functions-host

File.Move for Linux dedicated function apps results in lost telemetry

Open
#6,337 9 comments 0 reactions 2 assignees Assigned to @kashimiz View on GitHub
needs-investigation
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

Functions logs are written to file by LWAS to guarantee fault tolerance, and the files are periodically rotated so that we do not reach excessive sizes. Our log forwarding service - which is responsible for reading the log files and forwarding the data to our telemetry pipeline - listens for a specific inotify event associated with rotations. When it receives this event, it registers the log file as rotated and searches for the new log file. The File.Move API that is being used to rotate our functions logs is not emitting the correct inotify event, resulting in the log forwarder not being signaled to look for the new logs. By default, our platform initiates periodic restarts of the log forwarding service, which mitigates this issue. However, it can still cause log loss in high volume workloads.

Problem method is [LinuxAppServiceFileLogger.RollFiles()](https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script.WebHost/Diagnostics/LinuxAppServiceFileLogger.cs#L146).

Solution: Use system calls to make sure that the inotify event is emitted.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.