factor BaseTokenStream random docs generation into LineDocs [LUCENE-3975]

We improved on our 'random document generation' a lot in #4984

In fact these random docs find a lot of real bugs. Also the linedocs
driven from real data is also improved in the analyzer tests: it takes
substrings of random linedocs and makes 'partial docs'.

Really we should refactor this so that LineDocs uses a mix of real,
partial-real, and synthetic docs just like the analyzer tests.

This would help tests like term dictionary tests which are basically
static (even though they are random, the amount of documents is limited).

BaseTokenStreamTestCase would simply pull from LineDocs at that point,
but other tests would immediately see the benefits.

---
Migrated from [LUCENE-3975](https://issues.apache.org/jira/browse/LUCENE-3975) by Robert Muir (@rmuir)

Start by reading LineDocs and BaseTokenStreamTestCase, then inspect the analyzer tests' handling of real, partial-real, and synthetic documents. Trace how term dictionary tests currently obtain random documents. Done means BaseTokenStreamTestCase draws from the shared LineDocs approach and the affected tests receive the mixed document sources.

Written by the indexing model from the issue text.