klmr / klmr/box

Keep secondary storage cache of installed modules

Open
#264 0 comments 0 reactions 0 assignees View on GitHub
⚙️ enhancement ✨ new feature 💬 discussion
Dominant language
R
Stars
979
Forks
49
PR merge metrics
No merged PRs in 30d

Description

Currently loading a module that isn’t already in cache parses and evaluates all its source files, which is potentially time-consuming, especially when compared to loading a package: installed R packages aren’t loaded from source. Instead they’re loaded from a [lazy-load database](https://cran.r-project.org/doc/manuals/r-release/R-ints.html#Lazy-loading).

‘box’ could maintain a secondary storage cache (unless disabled) that is queried before the source version of a module is loaded, unless the latter has a more recent timestamp. In that case, the cache would be invalidated, the source version loaded, and subsequently cached.

R doesn’t seem to provide a public API for generating lazy-load databases, but I don’t understand the purpose of lazy loading for exported names anyway — using RDS with a custom serialisation hook for package/module dependencies seems easier.

Lastly, keeping modules cached also means we can finally implement byte-compilation of modules without a prohibitive overhead on loading.

Some notes:

* Cache path: `box.cache` (overridden by `R_BOX_CACHE`)
* defaults to `XDG_CACHE_HOME/R/%v/%p/box` (placeholders as for `R_LIBS_*`) or equivalent
* explicitly set to `NULL` to disable
* Is a modification timestamp sufficient to establish cache validity or is a hash required?
* Figure out how to customise serialisation of dependencies.
* Terminology in API: term “cache” is now overloaded because we unfortunately already have the function `purge_cache`.
* Cache module help as well?
* What about integration of compiled native code?
* Hook to run on “installation” of a module into the cache? (see #163)
* #14
* Add exported function to explicit add/remove modules to/from cache (e.g. `install`/`uninstall`)?
* As an included module? As a command line utility?

Contributor guide

Open the contributing guide

Research direction

The issue names no source files, tests, or entry points. Start by tracing the existing purge_cache and module-loading behavior, then review the proposed box.cache and R_BOX_CACHE settings; the scope is complete only when cache invalidation, serialization, disabling, and module loading behavior are defined and tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
performance, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.