HermitCrab: measure AnalysisStateKey key-space growth from ungated realizational unapplication counts

Summary

AnalysisStateKey keys every analysis state on a multiset of per-rule unapplication counts. For affix-process and compounding rules that count is bounded by the rule's MaxApplicationCount, because the gate that reads it sits in front of the writer. Realizational affix-process rules have no such gate, so their counts are unbounded in principle, and every distinct count value yields a distinct key.

Nothing here measures how far those counts actually climb, on either consumer of the key. That is the gap this issue tracks.

This is a measurement request, not a bug report. No incorrect output is claimed.

Who is affected: Machine and PanGloss maintainers changing AnalysisStateKey or analysis-state merging, and users with realizational (inflectional-template) grammars.

Why it matters on the default path

AnalysisStateKey has two consumers, and the unmeasured one is on by default:

Consumer	Site	Enabled
Analysis-cascade and template-battery memo	MemoizedCombinationRuleCascade.cs:42, AnalysisStratumRule.cs:205	opt-in (MaxDegreeOfParallelism = 1)
MergeEquivalentAnalyses	AnalysisStratumRule.cs:156	on by default (Morpher.cs:63)

Key-space growth has a different cost on each. On the memo it means more resident entries and earlier arrival at MaxMemoEntries, past which AnalysisScope.Store silently stops recording with no eviction. On the merge path it means fewer states merge, so more alternatives are carried forward — and that path is live in the default configuration.

The asymmetry

Rule type	Count gate before the writer	Writer	Bounded?
AffixProcessRule	AnalysisAffixProcessRule.cs:45 — GetUnapplicationCount(_rule) >= _rule.MaxApplicationCount	AnalysisAffixProcessRule.cs:62	yes, by MaxApplicationCount
CompoundingRule	AnalysisCompoundingRule.cs:46 — same test	AnalysisCompoundingRule.cs:133	yes, by MaxApplicationCount
RealizationalAffixProcessRule	none	AnalysisRealizationalAffixProcessRule.cs:56	unknown

Word.MorphologicalRuleUnapplied (Word.cs:339) increments the count for every rule including realizational ones; it only skips them for the _mruleApps trail (Word.cs:344). AnalysisStratumRule's own comment at the merge call site already notes the consequence — two words can differ in key while being Word.ValueEquals, precisely because UnappliedRuleCounts counts realizational rules that never enter _mruleApps.

There is a plausible bound: realizational unapplication requires _rule.RealizationalFeatureStruct.Unify(input.RealizationalFeatureStruct, out _) to succeed (AnalysisRealizationalAffixProcessRule.cs:46), and once a feature is realized, re-unification may fail. That is a hypothesis about feature-structure monotonicity, not something that has been checked.

Evidence so far, and its limits

While measuring a proposed key change on hc/memo-key-saturation (see the write-up at docs/hc-memo-key-saturation-measurement.md), a probe on the key constructor and a second at Word.MorphologicalRuleUnapplied recorded, on the two capped rule types:

• Conformance suite, 36 fixtures: 7,025 capped-rule key checks, 1,895 unapplications, 0 counts above cap.
• Sena, 60 words, memo on: 1,986,054 key checks, 454,647 unapplications, 0 counts above cap.
• Cap census over Sena, Mbugwe, en-hc and sp-hc: 1,443 capped rules, none with MaxApplicationCount above 1.

The probe deliberately returned early for rule types with no cap, so it collected nothing on realizational rules. The numbers above say only that the bounded types stay bounded. The unbounded type is unmeasured.

Done when

On real grammars with realizational morphology, and with MergeEquivalentAnalyses at its default:

1. The maximum and distribution of per-rule unapplication counts for RealizationalAffixProcessRule are known, per grammar and per rule.
2. Distinct resident key counts are known for both consumers. Memo.Count/TemplateMemo.Count are not currently exposed, and MemoInserts cannot substitute — AnalysisScope.Store assigns with table[key] = ..., so re-storing a live key counts twice.
3. The share of key-space attributable to realizational counts is separated from the rest of the key, by re-keying with that component held constant and comparing distinct-key counts.
4. Either a bound on realizational counts is established from feature-structure monotonicity and documented on AnalysisStateKey, or the absence of a bound is recorded with the measured worst case.

If a bound is established, saturating realizational counts becomes a candidate for the first time. If there is no bound, the same question applies to ReplayOnto's documented invariant at Word.cs:333 — equal unapplication multisets imply equal trail lengths — which holds today only because realizational counts are excluded from the trail on both sides of any comparison.

Out of scope

Merge correctness, which is #505. Memo resource budgets and eviction, which #485 covers for calibration. This issue is about how large the key space gets and what drives it.

Start with AnalysisStateKey consumers at MemoizedCombinationRuleCascade.cs:42, AnalysisStratumRule.cs:156 and :205, then inspect the count writer in AnalysisRealizationalAffixProcessRule.cs:56 and Word.cs:339. Measure realizational count distributions and distinct resident keys on real grammars with default merging, separating realizational contributions and documenting either a bound or the measured worst case.