llnl / llnl/scr

watchdog expects flush file to list a single location

Open
#413 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
108
Forks
37
PR merge metrics
No merged PRs in 30d

Description

The watchdog script uses the ``scr_flush_file`` command to query the location of the latest dataset:
https://github.com/LLNL/scr/blob/552ad3ec8e74a508f567a1bab0c0e2f278ea431d/scripts/common/scr_watchdog.in#L79

It expects a single value back, which it then compares to ``SYNC_FLUSHING``:
https://github.com/LLNL/scr/blob/552ad3ec8e74a508f567a1bab0c0e2f278ea431d/scripts/common/scr_watchdog.in#L103

The ``scr_flush_file`` command just happens to return the first location, which could be arbitrary:
https://github.com/LLNL/scr/blob/552ad3ec8e74a508f567a1bab0c0e2f278ea431d/src/scr_flush_file.c#L519

However, there is a problem here in that the flush file may list more than one location for a dataset. For example, the dataset might be listed in ``CACHE`` and ``PFS`` simultaneously. So really a location query should return a list of locations, not just one. Alternatively, we should be testing whether a particular value is in the list of locations.

Additionally, especially now that we have various async flush methods, more than ``SYNC_FLUSHING`` can indicate that a dataset is being written to the parallel file system.

Contributor guide

Open the contributing guide

Research direction

Start with scripts/common/scr_watchdog.in around lines 79 and 103, then inspect src/scr_flush_file.c around line 519 to understand how locations are returned. Determine how the watchdog should handle multiple locations and asynchronous flush states; it is done when its checks correctly identify datasets being written to the parallel file system.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, shell
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.