Improved implementation of language models in lucene [LUCENE-5847]

The current implementation of language models in lucene is based on the paper "A Study of Smoothing Methods for Language Models Applied to Ad Hoc Information Retrieval" by Zhai and Lafferty ('01). Specifically, LMDiricheltSimilarity and LMJelinikMercerSimilarity use a normalized smoothed score for a matching term in a document, as suggested in the above mentioned paper.

However, lucene doesn't assign a score to query terms that do not appear in a matched document. According to the "pure" LM approach, these terms should be assigned with a collection probability "background score". If one uses the Jelinik Mercer smoothing method, the final result list produced by lucene is rank equivalent to the one that would have been created by a full LM implementation. However, this is not the case for Dirichlet smoothing method, because the background score is document dependent. Documents in which not all query terms appear, are missing the document-dependant background score for the missing terms. This component affects the final ranking of documents in the list.

Since LM is a baseline method in many works in the IR research field, I attach a patch that implements a full LM in lucene. The basic issue that should be addressed here is assigning a document with a score that depends on **all** the query terms, collection statistics and the document length. The general idea of what I did is adding a new getBackGroundScore(int docID) method to similarity, scorer and bulkScorer. Than, when a collector assigns a score to a document (score = scorer.score()) I added the backgound score (score=scorer.score()+scorer.background(doc)) that is assigned by the similarity class used for ranking.

The patch also includes a correction of the document length such that it will be the real document length and not the encoded one. It is required for the full LM implementation.

---
Migrated from [LUCENE-5847](https://issues.apache.org/jira/browse/LUCENE-5847) by Hadas Raviv, updated May 09 2016
Attachments: [LUCENE-2507.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5847/LUCENE-2507.patch)