dotnet / dotnet/runtime

FileSystemWatcher on Linux: event-processing thread exits silently after interleaved directory renames (unpaired IN_MOVED_FROM handling) - watcher stops receiving events, no Error raised.

Open
#133,420 5 comments 0 reactions 0 assignees View on GitHub
area-System.IO bug
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

**EDIT:** original text described a shared-instance design (read from main, which is the .NET 11 implementation); architecture claims removed - see comments.

### Description

On .NET 10.x (Linux), an armed `FileSystemWatcher` is serviced by an inotify fd and a processing thread named `".NET File Watcher"`.

Under an easily produced filesystem workload - two directory renames applied back-to-back where the names collide (`lib -> src` while `code -> lib`) - the kernel delivers rename halves whose `IN_MOVED_FROM`/`IN_MOVED_TO` cookies don't pair up in one read window. After reading such a pair, the processing thread removes a child watch (`inotify_rm_watch`) and then **exits cleanly** (`exit(0)`) a few ms later, closing the inotify fd - while live watches are still registered and the managed `FileSystemWatcher` is still armed (`EnableRaisingEvents == true`).

From that moment the watcher receives no further events, ever. **No `Error` event is raised, no exception surfaces** - the failure is indistinguishable from "nothing changed on disk". The affected watcher never recovers.

.NET 7 is immune on the same machine and workload.

### Reproduction Steps

Single-file console app, BCL only. It arms one `FileSystemWatcher` (`IncludeSubdirectories = true`) on a temp tree, runs the colliding-rename chain in tight bursts alongside light background file writes, and checks liveness with a canary file at the watched root. It prints its own `/proc` post-mortem (inotify fd count + presence of the `".NET File Watcher"` thread).

```
dotnet publish -f net10.0 -r linux-x64 --self-contained -o out-net10
./out-net10/fswkill # dies within seconds; exit 2
dotnet publish -f net7.0 -r linux-x64 --self-contained -o out-net7
./out-net7/fswkill # control; survives 5000 rounds; exit 0
```

fswkill.csproj

```xml


Exe
net7.0;net10.0
fswkill
disable
disable

```

Program.cs (full source)

```csharp
// fswkill - reproducer for the .NET 10 Linux FileSystemWatcher silent death
// under interleaved (name-colliding) directory renames.
// Exit codes: 0 = survived, 2 = watcher died silently, 3 = died but Error fired.

using System;
using System.IO;
using System.Threading;

class Program
{
static long sEvents;
static volatile string sLastCanarySeen = "";
static volatile bool sErrorFired;

static int Main(string[] args)
{
int maxRounds = args.Length > 0 ? int.Parse(args[0]) : 5000;
string root = Path.Combine(
args.Length > 1 ? args[1] : Path.GetTempPath(),
"fswkill-" + Environment.ProcessId);

Console.WriteLine("runtime: " + Environment.Version + " | root: " + root);

string code = Path.Combine(root, "code");
string lib = Path.Combine(root, "lib");
string src = Path.Combine(root, "src");

Directory.CreateDirectory(Path.Combine(code, "child"));
Directory.CreateDirectory(Path.Combine(lib, "child"));
Directory.CreateDirectory(Path.Combine(root, "beacon"));
File.WriteAllText(Path.Combine(code, "child", "f.txt"), "x");
File.WriteAllText(Path.Combine(lib, "child", "f.txt"), "x");

using FileSystemWatcher fsw = new FileSystemWatcher(root);
fsw.IncludeSubdirectories = true;
fsw.NotifyFilter = NotifyFilters.FileName | NotifyFilters.DirectoryName
| NotifyFilters.LastWrite | NotifyFilters.CreationTime | NotifyFilters.Size;

FileSystemEventHandler onAny = delegate (object s, FileSystemEventArgs e)
{
Interlocked.Increment(ref sEvents);
if (e.Name != null && e.Name.StartsWith("canary_"))
sLastCanarySeen = e.Name;
};
fsw.Created += onAny;
fsw.Changed += onAny;
fsw.Deleted += onAny;
fsw.Renamed += delegate { Interlocked.Increment(ref sEvents); };
fsw.Error += delegate (object s, ErrorEventArgs e)
{
sErrorFired = true;
Console.WriteLine("!! Error event fired: " + e.GetException());
};
fsw.EnableRaisingEvents = true;

// background lock-file-style traffic keeps the inotify reads chunked
bool stop = false;
Thread writer = new Thread(delegate ()
{
string beacon = Path.Combine(root, "beacon", "lck");
int i = 0;
while (!Volatile.Read(ref stop))
{
try { File.WriteAllText(beacon, i++.ToString()); } catch { }
Thread.Sleep(1);
}
});
writer.IsBackground = true;
writer.Start();

Console.WriteLine("baseline: " + Vitals());

for (int round = 1; round <= maxRounds; round++)
{
// the colliding rename chain (name reuse is the essential ingredient)
try
{
Directory.Move(lib, src);
Directory.Move(code, lib);
Directory.Move(lib, code);
Directory.Move(src, lib);
}
catch (Exception ex)
{
Console.WriteLine("rename hiccup at round " + round + ": " + ex.Message);
EnsureLayout(code, lib, src);
Thread.Sleep(20);
}

if (round % 25 != 0)
continue;

if (!CanaryDelivered(root, round))
{
Volatile.Write(ref stop, true);
Console.WriteLine("== VERDICT: watcher DEAD at round " + round
+ " | events delivered: " + Interlocked.Read(ref sEvents)
+ " | Error fired: " + sErrorFired);
Console.WriteLine("post-mortem: " + Vitals()
+ " (watcher still armed: " + fsw.EnableRaisingEvents + ")");
return sErrorFired ? 3 : 2;
}
}

Volatile.Write(ref stop, true);
Console.WriteLine("== VERDICT: SURVIVED " + maxRounds + " rounds | events delivered: "
+ Interlocked.Read(ref sEvents) + " | " + Vitals());
return 0;
}

static bool CanaryDelivered(string root, int round)
{
string name = "canary_" + round;
try { File.WriteAllText(Path.Combine(root, name + ".txt"), "alive?"); }
catch (Exception ex) { Console.WriteLine("canary write failed: " + ex.Message); }

for (int waited = 0; waited < 2000; waited += 10)
{
if (sLastCanarySeen.StartsWith(name))
return true;
Thread.Sleep(10);
}
return false;
}

static void EnsureLayout(string code, string lib, string src)
{
try
{
if (!Directory.Exists(lib) && Directory.Exists(src)) Directory.Move(src, lib);
if (!Directory.Exists(code) && Directory.Exists(src)) Directory.Move(src, code);
if (!Directory.Exists(code)) Directory.CreateDirectory(Path.Combine(code, "child"));
if (!Directory.Exists(lib)) Directory.CreateDirectory(Path.Combine(lib, "child"));
}
catch { }
}

static string Vitals()
{
if (!OperatingSystem.IsLinux())
return "(vitals: non-linux)";

int inotifyFds = 0;
try
{
foreach (string fd in Directory.GetFiles("/proc/self/fd"))
{
try
{
FileSystemInfo target = new FileInfo(fd).ResolveLinkTarget(false);
if (target != null && target.FullName.Contains("inotify"))
inotifyFds++;
}
catch { }
}
}
catch { }

int watcherThreads = 0;
try
{
foreach (string task in Directory.GetDirectories("/proc/self/task"))
{
try
{
if (File.ReadAllText(Path.Combine(task, "comm")).Contains(".NET File Watch"))
watcherThreads++;
}
catch { }
}
}
catch { }

return "inotify fds: " + inotifyFds + ", '.NET File Watcher' threads: " + watcherThreads;
}
}
```

### Expected behavior

Events keep flowing for the still-registered watches - or, if event processing must stop, Error is raised on the affected FileSystemWatcher so the consumer can recover. A silent permanent stop while the watcher is armed should not be possible.

### Actual behavior

```
$ ./out-net7/fswkill # runtime 7.0.20, same machine
== VERDICT: SURVIVED 5000 rounds | events delivered: 21535 | inotify fds: 1, '.NET File Watcher' threads: 1
exit: 0

$ ./out-net10/fswkill # runtime 10.0.11
baseline: inotify fds: 1, '.NET File Watcher' threads: 1
== VERDICT: watcher DEAD at round 125 | events delivered: 456 | Error fired: False
post-mortem: inotify fds: 0, '.NET File Watcher' threads: 0 (watcher still armed: True)
exit: 2
```

strace of the processing thread's final moments:

```
21037 20:56:43.795823 read(38, "\4\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0lck\0\0\0\0\0\0\0\0\0\0\0\0\0", 8192) = 32
21037 20:56:43.795988 read(38, "\1\0\0\0@\0\0@\31n\1\0\20\0\0\0lib\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 64
21037 20:56:43.797961 inotify_add_watch(38, "/tmp/fswkill-21029/src", IN_MODIFY|IN_ATTRIB|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_ONLYDIR|IN_DONT_FOLLOW|IN_EXCL_UNLINK) = 2
21037 20:56:43.798056 read(38, "\4\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0lck\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 192
21037 20:56:43.798330 inotify_add_watch(38, "/tmp/fswkill-21029/lib", IN_MODIFY|IN_ATTRIB|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_ONLYDIR|IN_DONT_FOLLOW|IN_EXCL_UNLINK) = 2
21037 20:56:43.799258 inotify_rm_watch(38, 3) = 0
21037 20:56:43.799470 inotify_rm_watch(38, 2) = 0
21037 20:56:43.803517 exit(0) = ?
21037 20:56:43.803648 +++ exited with 0 +++
```

- The `lck` reads are the reproducer's background writer (`IN_MODIFY`).
- The 64-byte read on the root watch (wd 1) carries `IN_MOVED_FROM|IN_ISDIR`, cookie 0x16E19, name `lib` - plus a second event in the same read (strace truncates the payload display): the interleaved rename pair.
- The `inotify_add_watch` calls that follow are the watcher re-adding watches for the moved directories while still processing.
- Then two `inotify_rm_watch` calls and, 4 ms after the fatal read, `exit(0)` - clean, voluntary, no signal. No further reads on the fd ever; the process keeps running (the reproducer's verdict line prints afterwards), the watcher is still armed, and `Error` never fires..

By elimination on the exit path (no exception or Error, no overflow event, the rm_watch immediately preceding), the thread's exit follows directly from the unmatched-MOVED_FROM handling.

### Regression?

Yes. .NET 7 (7.0.20) is immune on the same machine and workload (per-instance fd + thread design). Reproduced on 10.0.10 and 10.0.11 (latest servicing). .NET 8/9 not tested.

### Known Workarounds

None satisfying. We currently detect the death with an out-of-band liveness probe (touch a watched path, require the event within a timeout) and rebuild the watcher - there is no callback-based way to notice it.

### Configuration

- .NET 10.0.10 and 10.0.11, linux-x64, self-contained deployments
- Ubuntu x86_64 (also reproduced on Amazon EC2 images), ext4
- Not architecture-specific as far as we know; Windows unaffected

### Other information

Real-world impact: we ship a version control system; a merge that applies "both sides moved directories" produces exactly this rename pattern in users' working trees, silently killing change detection for every workspace open in the process.

Contributor guide

Open the contributing guide

Research direction

Start by running the single-file reproducer in Program.cs with fswkill.csproj on Linux, comparing net10.0 with net7.0 and observing the watcher thread and inotify descriptors. Trace the Linux FileSystemWatcher event-processing path for interleaved IN_MOVED_FROM events; done means the watcher continues delivering events or raises Error instead of silently stopping.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, linux
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.