hyperledger-firefly / hyperledger-firefly/common

fswatcher Should Dedup Writes or Detect "CloseWrites"

Open
#156 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5
Forks
15
Avg merge
3d 6h
Merged PRs (30d)
4

Description

Following up from #149 - it's really a bit of rabbit hole if you look at https://github.com/fsnotify/fsnotify/issues/635 and the numerous other related issues.

In short, the comment for the `fsnotify.Write` event describes the issue:
```
// The pathname was written to; this does *not* mean the write has finished,
// and a write can be followed by more writes.
```

So what I've observed when using `os.WriteFile` on Linux systems + K8s PVCs is that initially `fswatcher` processes a `fsnotify.Write` event where the file was rewritten / has size of 0 bytes. It then receives a subsequent `fsnotify.Write` and the file is finished being written with a non-zero size. The problem is both the 0 bytes and the non-zero bytes events change the tracked hash, causing two unnecessary `onChange` events in the case where the file did not actually change, and one extra `onChange` event when the file did change. I've had to add a defense in my use of this utility to ignore `onChange` if the size of the file is 0 for now (which still isn't great because of the second `onChange`). For example from my logs:

```
Config file change detected. Event=WRITE Name=/data/peers.json Size=0 Hash=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Config file change detected. Event=WRITE Name=/data/peers.json Size=3687 Hash=2926b16fe9c31543f05e269eff2bd7d6542050923c348b1e13c6110b3c667e71
```

There's examples of how we could go to the trouble deduping the writes: https://github.com/fsnotify/fsnotify/blob/main/cmd/fsnotify/dedup.go
But that feels a bit convoluted if we can avoid it (sorry Windows).

Instead it looks like the maintainer is actively working on support non-portable events for Linux systems like `CLOSE_WRITE`: https://github.com/fsnotify/fsnotify/blob/main/fsnotify.go#L211-L219

These features are still in main of the lib / unreleased, but once they are we can detect if such events are supported and change what events we listen for as a result - make Linux optimized for detecting file changes which is the main OS targetted for production use cases of this library since it is used primarily in containerized applications.

This issue to track the `fsnotify` upstream for the more efficient event types and fix this once they are released. If too much time passes, we can consider maintaining an `fsnotify` fork.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.