apache / apache/lucene

Codec classes/packages that do not provide (automatic) file format back compat need to be more explicit about this in javadocs [LUCENE-5125]

Open
#6,189 5 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

rmuir noted in #6185...

> Currently (as documented), we don't provide index back compat for experimental codecs in lucene-codecs.jar.

...but except for a solr wiki page and solrconfig.xml comment, it's extremely non-obvious that any of these codec classes don't provide index backcompat.

- the codec module overview.html page describes the module as "Collection of useful codec, postings format and terms dictionary implementations" – with no indication that by using these "useful" implementations, the user gives up index backcompat.
- the package.html files in the individual packages of the codec module (appending, blockterms, bbloom, diskdv, etc...) also say nothing about index backcompat
- the individual classes in these codecs are mostly labeled with ``@lucene`.experimental` but in the resulting javadoc that merely says that "WARNING: This _API_ is experimental and might change in incompatible ways in the next release". Lots of classes in Lucene have this warning on them about their API (including the abstract codec apis themselves in lucene-core: DocValuesFormat, PostingsFormat, etc...) and that annotation (as far back as i can remember) has always only refered to the java API of the labeled class – never to whether using that class ment you were giving up on index format back compat.

Given how much effort and work is put into ensuring good index backcompat for default codec, we should be extremely explicit when/if alternative codecs do not support backcompat, so we don't frustrate/confuse users and leave them with the impression that they can never count on index backcompat just because they may not realize they were using an "unsupported" format option because of a blog post they read or advice they got on the mailing list about how to make something faster or use less ram.

---
Migrated from [LUCENE-5125](https://issues.apache.org/jira/browse/LUCENE-5125) by Chris M. Hostetter (@hossman), updated Sep 11 2014
Linked issues:
- #7002

Contributor guide

Open the contributing guide

Research direction

Start with the codec module overview.html and the package.html files for the individual codec packages, then review the relevant codec class Javadocs and their existing experimental annotations. Update the documentation so users can clearly identify which implementations do not provide index backcompat, and verify the generated Javadocs communicate that limitation consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.