Backport remaining Lucene 4.8.1 bug fixes

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Task description

We have effectively been targeting 4.8.1 instead of 4.8.0 for a while now, so we might as well close that gap before exploring #1293. This release includes [some worthwhile bugfixes](https://lucene.apache.org/core/4_8_1/changes/Changes.html).

What follows is the analysis by Claude Code against the Lucene 4.8.1 source code and git history. There may be errors/omissions.

## Background

Lucene.NET targets Lucene 4.8.1, but an audit of the upstream `4.8.0 → 4.8.1`
diff (the 16 `LUCENE-*` bug fixes in `lucene/CHANGES.txt`) found that several
fixes were never ported. No existing issue tracks these. #1293 is explicitly
scoped to fixes *after* 4.8.1 (4.9-4.10), so this gap is currently untracked.

The fixes below match pre-fix 4.8.0 behavior in current `master`. They are
grouped into batches that can be broken out into sub-issues. Several are
trivial; one (LUCENE-5628) fixes a real `StackOverflowException` that is
**already biting the port** (`AnalyzingSuggesterTest.TestTooLongSuggestion`
sits commented out with a "currently produces a stack overflow" note).

## Already ported (no action)

LUCENE-5662, LUCENE-5599/5600, LUCENE-5653, LUCENE-5559, LUCENE-5644,
LUCENE-5671 are all present, with matching tests for 5653, 5559, and 5599/5600.

## Remaining work

### Batch 1 — Trivial production fixes (1-5 lines each)

PR: #1413

- [X] **LUCENE-5635**: move `stream.Reset()` inside the `try` block in
`DocInverterPerField.cs` (~line 107) so an exception on reset is handled.
- [X] **LUCENE-5668**: change `eligible.Count >= allowedSegCountInt` to `>` in
`TieredMergePolicy.cs` (~line 415); port the `testUnbalancedMergeSelection`
test.
- [X] **LUCENE-5656**: move `dvGens.Add(gen)` inside the per-gen loop in
`SegmentReader.cs` (~lines 197-209) instead of `AddRange` after the loop
(rare fd leak on docvalues-update exception).
- [X] **LUCENE-5644 follow-up**: add the missing `PulseAll`/`notifyAll` at the
end of `DeactivateUnreleasedStates()` in `DocumentsWriterPerThreadPool.cs`.

### Batch 2 — LUCENE-5641 (RateLimiter)

PR: #1420

- [X] Rewrite `RateLimiter.cs`: add `MinPauseCheckBytes`, synchronize `Pause`,
drop `nsPerByte`/`lastNS` and the `bytes == 1` shortcut.
- [X] Update `RateLimitedIndexOutput.cs` to accumulate `bytesSinceLastPause`
and only call `Pause` when it exceeds `MinPauseCheckBytes`.
- [X] Port the multithreaded `TestRateLimiter.testThreads`.

Fixes the "silently rate-limits at 8 MB/sec regardless of configured rate" bug.

### Batch 3 — LUCENE-5639 (Token.PositionLength)

PR: #1421

- [X] Wire `positionLength` into all 9 sites in `Token.cs`: `Clear`, `Clone`,
`Equals`, `GetHashCode`, `ClearNoTermBuffer`, the private `Reinit`, the three
`Reinit(Token …)` overloads, `CopyTo`, and `ReflectWith`. (The field/property
exist but none of the handling is wired up.)
- [X] Add the `positionLength` attribute-map entry in `TestToken`.

### Batch 4 — LUCENE-5628 + LUCENE-5660 (highest priority)

PR: #1422

- [X] **LUCENE-5628**: make `GetFiniteStrings` iterative (PathNode stack) in
`SpecialOperations.cs` instead of recursive (one stack frame per character).
- [X] **LUCENE-5660**: re-enable the suggester guard in `AnalyzingSuggester.cs`
(currently `Debugging.Assert(SpecialOperations.IsFinite(...))` at ~line 982).
- [X] **OfflineSorter**: add the `len > short.MaxValue` guard in
`ByteSequencesWriter.Write` (`OfflineSorter.cs`).
- [X] Add the `GetFiniteStringsRecursive` helper to `AutomatonTestUtil.cs`.
- [X] Port the full 4.8.1 `TestSpecialOperations` suite and un-comment
`AnalyzingSuggesterTest.TestTooLongSuggestion` (`AnalyzingSuggesterTest.cs`
~line 1502).

Resolves an active `StackOverflowException` already stubbed out in the suite.

### Batch 5 — LUCENE-5654 + test infrastructure (largest; gated by infra)

PR: #1424

- [x] **Production**: convert the 4 close() call sites to the success-boolean
pattern: `CompoundFileDirectory`, `CompoundFileWriter`, `Directory.Copy`, and
`FSDirectory` output close. (Keep the existing
`DisposeWhileHandlingException(priorException, …)` IOUtils overloads — only
the call sites change.)
- [x] **Test framework prerequisite**: port the entire **cranky codec package**
(9 files: `CrankyCodec` + `CrankyDocValues/FieldInfos/LiveDocs/Norms/
Postings/SegmentInfo/StoredFields/TermVectorsFormat`). Only `CrankyTokenFilter`
is ported today.
- [x] **Tests** (blocked on the two prerequisites above): port
`TestIndexWriterExceptions2`, `TestIndexWriterOutOfMemory`, and
`TestIndexWriterThreadsToSegments` (the last validates LUCENE-5644).
- [ ] **LUCENE-5644 follow-up** (`011a35b0a`, no JIRA): add active-thread-state counting + infoStream message to `FindLargestNonPendingWriter` in `FlushPolicy.cs` (~line 131) and prefix the `FlushByRamOrCountsPolicy.cs` flush message with `"trigger "` (~line 100).

## Suggested sequencing

Batches 1-4 are independent and deliver all user-facing production fixes;
**Batch 4 first** since it resolves an active stack-overflow. Batch 5 is a
larger infra investment whose production payload (5654) is lower-risk, so it's
a reasonable last step.