TextSearch or StringSearch package

Continued from https://github.com/johnmyleswhite/TextAnalysis.jl/issues/13 ...

@malmaud , I see two possible scopes for this package. One would be to confine it to in-RAM string search algorithms like Aho-Corasick, Boyer-Moore, etc. This would be fairly straightforward, although one possible complication is that these algorithms really apply to any kind of abstract sequence, not just strings (of characters), so the API might need to be designed as such to be maximally useful. And at that point we are getting a little out of the realm of "text" processing.

The other direction this package could go is to include both searching of in-RAM strings and of text corpora, which might be in RAM or on disk. Other issues of true text search like lexing, thesauri, inverted indices, etc, would then arise eventually.

I think option #2 is the better design (although it is obviously more work). Some of my research interests lie in this direction so I could contribute a fair amount.

So I am open to preferences between these two choices or suggestions that I haven't considered. I'm also open to naming suggestions.

I am also curious about whether there is any kind of (even very rough) roadmap for the various text processing and basic NLP tasks and data structures that will form a common core for JuliaText, including for a potential TextSearch package. For example:
- Will there be common data structures for corpora, documents, sentences, tokens, part-of-speech tags, parse trees, etc, (which gives the API more structure), or will there just be strings and arrays of strings everywhere (which makes the API more accessible for newcomers)?
- Where (what package) will "shallow NLP" tasks such as sentence splitting, stemming, and tokenization be handled?

I'd propose that there should be a TextBase package, along the lines of StatsBase, that defines a few of these fundamental data structures and operations. Perhaps TextAnalysis could serve as inspiration for this, but I think things like TFIDF, LSA, and perhaps stemming, could be left to a different package.