Handling of missing and stray log files in Bluefors Agent
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 17
- Forks
- 20
- Avg merge
- 1h 22m
- Merged PRs (30d)
- 2
Description
While this somewhat relates to ongoing development in #174, the underlying issues should still be fixed, either in that PR or a subsequent one.
I'll start with this log segment:
2021-06-03T05:20:00+0000 /logs/21-06-03/maxigauge 21-06-03.sync-conflict-20210603-011948-I2BQPMU.log not yet open, opening...
2021-06-03T05:20:00+0000 /logs/21-06-03/Flowmeter 21-06-03.sync-conflict-20210603-011958-I2BQPMU.log not yet open, opening...
2021-06-03T05:20:00+0000 Not publishing stale data. Make sure your log file sync is done at a rate faster than once ever 2 minutes.
2021-06-03T05:20:00+0000 Not publishing stale data. Make sure your log file sync is done at a rate faster than once ever 2 minutes.
2021-06-03T05:20:00+0000 Not publishing stale data. Make sure your log file sync is done at a rate faster than once ever 2 minutes.
...
...many more of these...
...
2021-06-03T05:20:03+0000 Not publishing stale data. Make sure your log file sync is done at a rate faster than once ever 2 minutes.
2021-06-03T05:20:03+0000 Not publishing stale data. Make sure your log file sync is done at a rate faster than once ever 2 minutes.
2021-06-03T09:00:59+0000 acq:0 Crash in thread: [Failure instance: Traceback: <class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/logs/21-06-03/Flowmeter 21-06-03.log'
/usr/lib/python3.8/threading.py:932:_bootstrap_inner
/usr/lib/python3.8/threading.py:870:run
/usr/local/lib/python3.8/dist-packages/twisted/_threads/_threadworker.py:47:work
/usr/local/lib/python3.8/dist-packages/twisted/_threads/_team.py:181:doWork
--- <exception caught here> ---
/usr/local/lib/python3.8/dist-packages/twisted/python/threadpool.py:238:inContext
/usr/local/lib/python3.8/dist-packages/twisted/python/threadpool.py:254:<lambda>
/usr/local/lib/python3.8/dist-packages/twisted/python/context.py:118:callWithContext
/usr/local/lib/python3.8/dist-packages/twisted/python/context.py:83:callWithContext
bluefors_log_tracker.py:488:start_acq
bluefors_log_tracker.py:367:read_and_publish_logs
]
2021-06-03T09:00:59+0000 acq:0 Status is now "done".
There are two problems here:
- The Agent matches files that we don't want to open (the sync-conflict files, a product of a syncing software in use in the Cornell setup)
- The network file share, we believe, disappears briefly at some point, and the file we want to open goes missing. The monitoring process crashes when this happens.
The solutions would be to restrict the matching of filenames a bit more, and to catch and handle this exception properly, probably by logging a warning or error and trying again on the next loop.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in bluefors_log_tracker.py at start_acq (line 488) and read_and_publish_logs (line 367), then trace how log filenames are matched and opened. Reproduce the shown cases if possible; done means sync-conflict files are ignored and a temporary FileNotFoundError is logged and retried on a later loop instead of crashing the agent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100