`ConcurrentDictionary<TKey,TValue>` causes disproportionately high full GC pause times for large counts

I’m not sure whether this is expected due to the internal node-based/object-heavy structure of ConcurrentDictionary, but the magnitude surprised me.

There's a large GC-pause-time difference between `Dictionary` and `ConcurrentDictionary` under a benchmark that fills each collection with many entries and then forces a full blocking compacting Gen2 collection.

To demonstrate the issue, the code below uses:

* A `TKey` as a 16-byte struct (`Key128`) and a `TValue` as a 50-byte struct (`Value400`). This way, the standard Dictionary doesn't allocate objects per nodes as both key and values are structs.
* The collection is populated sequentially on a single thread, after population, a forced full Gen2 GC is measured
* `ConcurrentDictionary` consistently shows much higher GC time than `Dictionary` at the same logical entry count
* The gap becomes very large at 10M+ entries, and higher than expected from simply the number of objects to be traversed.

This may be expected to some degree because `ConcurrentDictionary` has more internal object structure, but the GC pause increase looks disproportionately large relative to the retained managed size increase, and I couldn't find any documentation related to this.

## Environment

The issue is reproducible in all OS and both server and workspace GC.

```text
.NET: 10.0.5
OS: Microsoft Windows 10.0.26200
Process architecture: X64
Server GC: False
Latency mode: Interactive
```

## Reproduction Code

```csharp
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Diagnostics;
using System.Runtime.InteropServices;

const int DefaultStart = 10;
const int DefaultMax = 200_000_000;
bool exactCollect = false;

int start = DefaultStart;
int max = DefaultMax;

foreach (var arg in args)
{
if (arg.StartsWith("--start=", StringComparison.OrdinalIgnoreCase) &&
int.TryParse(arg["--start=".Length..], out var parsedStart))
{
start = parsedStart;
}
else if (arg.StartsWith("--max=", StringComparison.OrdinalIgnoreCase) &&
int.TryParse(arg["--max=".Length..], out var parsedMax))
{
max = parsedMax;
}
else if (arg.StartsWith("--exactCollect=", StringComparison.OrdinalIgnoreCase) &&
bool.TryParse(arg["--exactCollect=".Length..], out var parsedExact))
{
exactCollect = parsedExact;
}
}

Console.WriteLine($".NET: {Environment.Version}");
Console.WriteLine($"OS: {RuntimeInformation.OSDescription}");
Console.WriteLine($"Process architecture: {RuntimeInformation.ProcessArchitecture}");
Console.WriteLine($"Server GC: {System.Runtime.GCSettings.IsServerGC}");
Console.WriteLine($"Latency mode: {System.Runtime.GCSettings.LatencyMode}");
Console.WriteLine($"Pointer size: {IntPtr.Size * 8}-bit");
Console.WriteLine($"Key128 size: {Marshal.SizeOf()} bytes");
Console.WriteLine($"Value400 size: {Marshal.SizeOf()} bytes");
Console.WriteLine($"Start count: {start:N0}");
Console.WriteLine($"Max count: {max:N0}");
Console.WriteLine($"Collect mode: {(exactCollect ? "GC.Collect(2)" : "GC.Collect(2, Forced, blocking:true, compacting:true)")}");
Console.WriteLine();

WarmUp(exactCollect);

var counts = BuildCounts(start, max);

Console.WriteLine("=== object[] baseline ===");
RunSeries(
"Object Array",
counts,
FillObjectArray,
exactCollect);

Console.WriteLine("=== Dictionary baseline ===");
RunSeries(
"Dictionary",
counts,
FillDictionary,
exactCollect);

Console.WriteLine();
Console.WriteLine("=== ConcurrentDictionary ===");
RunSeries(
"ConcurrentDictionary",
counts,
FillConcurrentDictionary,
exactCollect);

static List BuildCounts(int start, int max)
{
var counts = new List();
long n = start;
while (n <= max)
{
counts.Add((int)n);
n *= 10;
}

if (counts.Count == 0 || counts[^1] != max)
counts.Add(max);

return counts;
}

static void WarmUp(bool exactCollect)
{
var d = new Dictionary(1024);
var c = new ConcurrentDictionary(Environment.ProcessorCount, 1024);

var rng = new SplitMix64(123456789);
for (int i = 0; i < 1024; i++)
{
var k = Key128.Create(ref rng);
var v = Value400.Create(ref rng);
d[k] = v;
c[k] = v;
}

ForceFullGC(exactCollect);
GC.KeepAlive(d);
GC.KeepAlive(c);
}

static void RunSeries(
string label,
IReadOnlyList counts,
Func fill,
bool exactCollect)
{
Console.WriteLine(
$"{Pad("Entries", 14)}" +
$"{Pad("Fill ms", 14)}" +
$"{Pad("GC ms", 14)}" +
$"{Pad("Managed MB", 16)}" +
$"{Pad("Gen2 before", 14)}" +
$"{Pad("Gen2 after", 14)}" +
$"{Pad("Status", 16)}");

Console.WriteLine(new string('-', 102));

foreach (var count in counts)
{
try
{
ForceFullGC(exactCollect);

long managedBefore = GC.GetTotalMemory(forceFullCollection: true);
int gen2Before = GC.CollectionCount(2);

var fillSw = Stopwatch.StartNew();
var collection = fill(count);
fillSw.Stop();

long managedAfterFill = GC.GetTotalMemory(forceFullCollection: false);

var gcSw = Stopwatch.StartNew();
ForceFullGC(exactCollect);
gcSw.Stop();

int gen2After = GC.CollectionCount(2);
long managedAfterGC = GC.GetTotalMemory(forceFullCollection: false);

Console.WriteLine(
$"{Pad(count.ToString("N0"), 14)}" +
$"{Pad(fillSw.ElapsedMilliseconds.ToString("N0"), 14)}" +
$"{Pad(gcSw.ElapsedMilliseconds.ToString("N0"), 14)}" +
$"{Pad(((managedAfterGC) / (1024.0 * 1024.0)).ToString("N1"), 16)}" +
$"{Pad(gen2Before.ToString(), 14)}" +
$"{Pad(gen2After.ToString(), 14)}" +
$"{Pad("OK", 16)}");

GC.KeepAlive(collection);
GC.KeepAlive(managedBefore);
GC.KeepAlive(managedAfterFill);
GC.KeepAlive(managedAfterGC);
}
catch (OutOfMemoryException)
{
Console.WriteLine(
$"{Pad(count.ToString("N0"), 14)}" +
$"{Pad("-", 14)}" +
$"{Pad("-", 14)}" +
$"{Pad("-", 16)}" +
$"{Pad("-", 14)}" +
$"{Pad("-", 14)}" +
$"{Pad("OOM", 16)}");

ForceFullGC(exactCollect);
break;
}
catch (Exception ex)
{
Console.WriteLine(
$"{Pad(count.ToString("N0"), 14)}" +
$"{Pad("-", 14)}" +
$"{Pad("-", 14)}" +
$"{Pad("-", 16)}" +
$"{Pad("-", 14)}" +
$"{Pad("-", 14)}" +
$"{Pad(ex.GetType().Name, 16)}");
break;
}
}
}

static object[] FillObjectArray(int count)
{
var dict = new object[count];

for (int i = 0; i < count; i++)
{
dict[i] = new ();
}

return dict;
}

static Dictionary FillDictionary(int count)
{
var dict = new Dictionary(count);
var rng = new SplitMix64(0x1234_5678_9ABC_DEF0UL ^ (ulong)count);

for (int i = 0; i < count; i++)
{
var key = Key128.Create(ref rng);
var value = Value400.Create(ref rng);
dict.Add(key, value);
}

return dict;
}

static ConcurrentDictionary FillConcurrentDictionary(int count)
{
int concurrencyLevel = Environment.ProcessorCount;
var dict = new ConcurrentDictionary(concurrencyLevel, count);
var rng = new SplitMix64(0x0FED_CBA9_8765_4321UL ^ (ulong)count);

for (int i = 0; i < count; i++)
{
var key = Key128.Create(ref rng);
var value = Value400.Create(ref rng);
if (!dict.TryAdd(key, value))
throw new InvalidOperationException("Unexpected duplicate key generated.");
}

return dict;
}

static void ForceFullGC(bool exactCollect)
{
if (exactCollect)
{
GC.Collect(2);
}
else
{
GC.Collect(2, GCCollectionMode.Forced, blocking: true, compacting: true);
}

GC.WaitForPendingFinalizers();

if (exactCollect)
{
GC.Collect(2);
}
else
{
GC.Collect(2, GCCollectionMode.Forced, blocking: true, compacting: true);
}
}

static string Pad(string s, int width) => s.PadRight(width);

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public readonly struct Key128 : IEquatable
{
public readonly ulong A;
public readonly ulong B;

public Key128(ulong a, ulong b)
{
A = a;
B = b;
}

public static Key128 Create(ref SplitMix64 rng)
=> new Key128(rng.NextUInt64(), rng.NextUInt64());

public bool Equals(Key128 other) => A == other.A && B == other.B;

public override bool Equals(object? obj) => obj is Key128 other && Equals(other);

public override int GetHashCode()
{
var hc = new HashCode();
hc.Add(A);
hc.Add(B);
return hc.ToHashCode();
}

public override string ToString() => $"{A:X16}{B:X16}";
}

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public readonly struct Value400
{
public readonly ulong A;
public readonly ulong B;
public readonly ulong C;
public readonly ulong D;
public readonly ulong E;
public readonly ulong F;
public readonly ushort G;

public Value400(ulong a, ulong b, ulong c, ulong d, ulong e, ulong f, ushort g)
{
A = a;
B = b;
C = c;
D = d;
E = e;
F = f;
G = g;
}

public static Value400 Create(ref SplitMix64 rng)
=> new Value400(
rng.NextUInt64(),
rng.NextUInt64(),
rng.NextUInt64(),
rng.NextUInt64(),
rng.NextUInt64(),
rng.NextUInt64(),
(ushort)rng.NextUInt64());
}

public struct SplitMix64
{
private ulong _state;

public SplitMix64(ulong seed) => _state = seed;

public ulong NextUInt64()
{
ulong z = (_state += 0x9E3779B97F4A7C15UL);
z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9UL;
z = (z ^ (z >> 27)) * 0x94D049BB133111EBUL;
return z ^ (z >> 31);
}
}
```

## Results

```text
=== object[] baseline ===
Entries Fill ms GC ms Managed MB Gen2 before Gen2 after Status
------------------------------------------------------------------------------------------------------
10 0 0 0.1 5 7 OK
100 0 0 0.1 10 12 OK
1,000 0 0 0.1 15 17 OK
10,000 0 0 0.4 20 22 OK
100,000 0 2 3.1 25 27 OK
1,000,000 29 23 30.6 30 33 OK
10,000,000 441 238 305.2 36 42 OK
100,000,000 3,463 2,389 3,051.8 45 51 OK
200,000,000 6,223 4,951 6,103.6 54 57 OK

=== Dictionary baseline ===
Entries Fill ms GC ms Managed MB Gen2 before Gen2 after Status
------------------------------------------------------------------------------------------------------
10 0 0 0.1 60 62 OK
100 0 0 0.1 65 67 OK
1,000 0 0 0.1 70 72 OK
10,000 2 0 0.8 75 77 OK
100,000 5 0 8.3 80 82 OK
1,000,000 93 42 88.8 85 87 OK
10,000,000 1,967 74 763.0 90 92 OK
100,000,000 32,004 157 7,629.5 95 97 OK
200,000,000 72,399 197 15,258.9 100 102 OK

=== ConcurrentDictionary ===
Entries Fill ms GC ms Managed MB Gen2 before Gen2 after Status
------------------------------------------------------------------------------------------------------
10 0 0 0.1 105 107 OK
100 0 0 0.1 110 112 OK
1,000 0 0 0.2 115 117 OK
10,000 5 1 1.1 120 122 OK
100,000 18 12 10.0 125 127 OK
1,000,000 446 132 100.5 130 135 OK
10,000,000 11,141 1,502 1,068.2 138 145 OK
100,000,000 143,795 10,903 10,681.2 148 154 OK
200,000,000 253,100 41,531 21,362.4 157 162 OK
```

## Observed behavior

`ConcurrentDictionary` incurs dramatically higher post-fill full GC pause times than `Dictionary` and object[] for the same number of entries.

At larger sizes:

* **10M entries**

* `Dictionary`: 74 ms GC
* `ConcurrentDictionary`: 1,502 ms GC
* about **20x** slower

* **100M entries**

* `Dictionary`: 157 ms GC
* `ConcurrentDictionary`: 10,903 ms GC
* about **69x** slower

* **200M entries**

* `Dictionary`: 197 ms GC
* `ConcurrentDictionary`: 41,531 ms GC
* about **211x** slower

The retained managed size does increase for `ConcurrentDictionary`, but not nearly enough to explain the GC-time multiplier by size alone:

* **100M entries**

* `Dictionary`: 7,629.5 MB
* `ConcurrentDictionary`: 10,681.2 MB
* about **1.40x** memory

* **200M entries**

* `Dictionary`: 15,258.9 MB
* `ConcurrentDictionary`: 21,362.4 MB
* about **1.40x** memory

So GC time grows far faster than retained managed size.

I would have expected that `ConcurrentDictionary` has a higher steady-state memory usage than `Dictionary` with some extra object graph / metadata cost, but the full GC cost here appears much larger than the memory overhead alone would suggest. The test run on workspace GC, but the same behaviour is observed when running on server GC.

Is this level of GC pause amplification for `ConcurrentDictionary` expected, or is this something the runtime/libraries team would consider a performance issue worth investigating?