BibFormat: lazy record formatting

Originally on 2012-04-02

0) It is expensive to format records on the fly, especially on pages such as search results, that need to display many records on the same page. So `bibreformat` daemon is periodically running in the background and updates HTML brief and other formats and stores them in the `bibfmt` format cache, from which the search result page simply selects needed format to display it to the user. This is good for display speed in our "many-SELECT's-few-UPDATE's" database conditions.

(Historically speaking, `BibFormat` had PHP support for old formats written in PHP, so it was especially expensive to call the formatting engine from the search pages that were written in Python. Now that formats are done fully in Python, they are faster to produce.)

For production, after ingestion, it is good to show a record in its new state as soon as possible after ingestion, which is why we have introduced and implemented #870.

Both these goals can be achieved with lazy record formatting which will permit us to remove `bibreformat` daemon task from running and have always up-to-date formats.

1) It will be useful to introduce a new option to `format_record()` that will be named like `store_in_cache=True` that will proces given record ID and that will optionally store resulting format in bibfmt cache. When cache is up-to-date, and `on_thy_fly` is not `False`, it would read the cache to return the given format.

2) `bibfmt` table is growing big. It will be useful to decouple "important" formats such as MARCXML stored there by the upload, and "less important" formats such as HB that are simply only cached there and that can be recreated at any time from "important" formats.

So we can introduce a new table like `fmtCACHE` that would store all pre-cached formats that used to be generated by `bibformat`, such as HB, HD etc. The structure of `fmtCACHE` could look like that of `bibfmt`.
In the future it could store even parts of the formats in case some elements are dynamic and depend upon logged-in user credentials.

(Important metadata-related internal formats that were not generated by `bibreformat`, such as `recstruct`, would be left in `bibfmt`.)

3) Once we have lazy record formatting facility in place, `bibreformat` daemon would not have to be run as a bibsched task anymore. We could remove this capability and introduce new CLI options that would offer operating on the `fmtCACHE` table. E.g. delete pre-cached formats for collection_Poetry_ when `Poem.bft` template changes, see also #972. E.g. delete all HB formats because we change URL of the installation, see [wiki:HowTo/HowToChangeSiteUrl]. Etc.

Start by tracing the format_record() entry point, the bibfmt cache, and the bibreformat daemon or bibsched task. The proposal also mentions a new fmtCACHE table and CLI operations for invalidation; done would require lazy cache reads and writes, cache separation, daemon removal, and replacement CLI behavior.

Written by the indexing model from the issue text.