Azure / Azure/azure-linux-extensions
Diagnostic extension may lose logs due to out of sync between mdsd and omsagent
- Dominant language
- Python
- Stars
- 333
- Forks
- 278
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 4
Description
Hi,
I meet a problem when I use diagnostic extension to collect custom log files. After investigating I suspect that the reason is the mdsd uses out-dated timestamp sent by omsagent(fluentd).
In fluentd it will set a time stamp for each record. However, It may takes a bit longer time for fluentd to send out the log records if there are many lines of logs, such as reading from the beginning of the log file. In mdsd it will use this time stamp as the key ("PreciseTime") to store the log records. When mdsd "flush" the records in its local sink, it will calculate a "time window" (in diagnostic extension the interval is 60s). Mdsd only sends the records that locate in this "time window" and discards the older ones.
It's possible that fluentd accept an event and set timestamp to 00:04:59.990 but the mdsd receives the record at 00:05:00.500. In this case, when mdsd flushes the records between 00:05:00 and 00:06:00, this record won't be collected and it's lost.
I am not sure whether my suspicion is correct but from this PR [comment](https://github.com/Azure/fluentd-plugin-mdsd/pull/42#pullrequestreview-64670637) I think you have known this problem but there is no good solution (I cannot control the speed of generating logs). Do you have a plan to improve this?
Thanks.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.