[RFC] embedding files as OCaml modules ("crunch")

Need

In some cases, programs need to access data at runtime without accessing the filesystem. This RFC describes a mechanism to embed data into OCaml modules.

Approach

The proposed solution is to extend the actions language with a new action crunch:

Syntax

<action> ::= ...
        |  ( crunch <crunch-arg>+ )

<crunch-arg> ::= ( <string> <filename> )

Static semantics

• Every <filename> is recorded as a dependency to the action.
• Every <string> should be a valid OCaml identifier.
• Using this action requires a lower bound on the dune-lang version.

Dynamic semantics

The crunch action evaluates to a string which represents a valid OCaml structure. This structure has one top-level definition per ̀<crunch-arg>, which binds the contents of the file named <filename> to the identifier <string>.

Possible implementation

A possible technique is, for each file, to determine a separator sep which does not occur in it, and emit the following code:

let name = {sep|contents|sep}

Example

An application sets up a HTTPS library using a certificate and a key. These pieces of data are generated by another process, and to be embedded in a webserver process. The following creates an executable that does not require access to the files at runtime.

; dune
(rule
 (with-stdout-to data.ml
  (crunch
   (key key.pem)
   (cert cert.pem))))

(* server.ml *)
let tls_config = Tls_config.create ~key_pem:Data.key ~cert:Data.cert in
let server = Server.create ~tls_config in
Server.start server

Benefits

This pattern is common and several projects use it. Its design is simple and composes with the rest of the action language (file dependencies, output as string). It can be implemented separately from dune (see next section) but would add an external dependency for something that is a build system concern. Also, even if the dependency is just an executable, its build-dependencies would need to be installed in the same opam switch, and it would need to handle all the ocaml versions that dune supports.

Alternatives

Several implementations of this pattern are available:

• ocaml-crunch
• ocp-ocamlres

It is also something that can be encoded directly using a dune rule, for example:

https://github.com/ocaml/ocaml-lsp/blob/f2303a7e3e516a36ec09b182d06003ee845e8e38/lsp/bin/dune#L5-L11

Design questions

This section contains question and lists alternative designs that do not seem necessary.

Name

This document uses "crunch" in reference to ocaml-crunch but this might not be the best name to carry what this feature does. There is not much "crunching" as in compression or data processing, but the name is recognizable. Maybe embed or generate-data-module is more descriptive.

Generating mli files

It is possible to generate the corresponding mli file (with one val name : string per <crunch-arg>). That complicates the design because one action can only emit a single string. So, the action would need to write the files (as targets) by itself, for example:

Also, the main benefit in having mli files is performance of incremental builds (to avoid rebuilding all dependencies when the file contents change) but in development mode this kind of changes will cause an early cut in dune.

Sourcing data from arbitrary actions

The following would allow embedding the output of arbitrary actions:

<crunch-arg> ::= ( <string> <action> )

However, this is not more powerful since it is always possible to add a rule to write the output of an action to a file, and embed this generated file.

This also keeps the common case shorter by removing the need to use (cat).

Restricting the action to one file

The action can be simplified to embed a single file. To embed several files, one would use (progn) to generate a module:

(with-stdout-to
 data.ml
 (progn
  (crunch (a a.txt))
  (crunch (b b.txt))))

But the use case to embed several files seems common enough to justify the shorter syntax. In addition, some future extensions could require inserting a prelude which would make the law (crunch A B) == (progn (crunch A) (crunch B)) not hold anymore.

Dynamic aspects

ocaml-crunch and ocp-ocamlres expose a dynamic interface where instead of exposing one value per file, a single function is exposed, which is invoked (at runtime) with the name of the embedded file.

The dynamic version can deal with list of files that is not known statically; but on the other hand the static version is total. Also, it is possible to encode a dynamic version by generating a dispatch module using the static version and the list of files.

Extensions

This section contains possible future extensions for this feature (or could reasonably be part of the first version).

Name inference

It is possible to allow a shorthand version of <crunch-arg> where the name is implicit:

<crunch-arg> ::= ...
              |  <filename>

There are several ways to define it, either define a slugification function that converts a filename to a valid identifier, or restrict the set of filenames that allow that.

Beyond strings

While embedding strings fits most use cases, some libraries internally use Bigarray or Cstruct. To convert into the right type, a layer of let x = Cstruct.of_string Data.x in a separate module can work, but the compiler might not be able to eliminate the string copy.

If that is a concern, this feature could be extended to insert a call to a runtime constructor (Cstruct.of_string) or to a compile-time transformation (let%cstruct = "...") to lift that transformation to a preprocessor.

A related feature is parsing structured file formats at compile time: let%json x = "..." could replace itself by the corresponding JSON AST.

Start with the RFC's proposed crunch action syntax, static semantics, and dynamic semantics, then review the listed design questions about naming, interfaces, and single versus multiple files. Compare the existing dune rule example and the ocaml-crunch and ocp-ocamlres alternatives. Done means the design is settled and the action embeds named file contents as a valid OCaml module.