Vector kernel for Jupyter

Development of new Vector configs can be greatly facilitated by Vector kernel for Jupyter.

To skip the introduction, there is an example of how notebooks using such kernel could have looked like: Vector notebook.

Required features

The kernel should allow to write one or multiple components using Vector's TOML syntax in Jupyter cells and then evaluate these cells. The evaluated components should be added to the topology of current running Vector instance. If components with the same names already exist in the topology, they are expected to be automatically replaced.

If the last component in a cell is source or transform, then one (or, if the user specifies it, multiple) events are pulled from it by the kernel automatically and displayed to the user in the notebook. If the last component is a sink, then nothing is displayed, it is just added to the topology in the currently running kernel.

Compatibility

Implementing the kernel would make it possible to use it from Jupyter Notebook, Jupyter Lab (a newer generation of Jupyter Notebook interface), and IPython (console version of the two above).

Proposed implementation

The easiest way to implement this is using Jupyter library for wrapper kernels. With this approach no modifications in Vector code are required, because all REPL-related logic can be implemented in external Python code.

The data flow can be the following:

• When the kernel is started, it waits for cells code which it can evaluate.
• After the first cell code comes to the kernel, it spawns a new Vector instance with temporary config file containing the code of the components from the cell plus a console or socket sink connected to the last component (only in case of last component is source or transform).
• Then the kernel waits for output from the appended sink and displays it to the user as result of evaluation of the automatically created sink, otherwise it reports errors from Vector's log.
• On subsequent evaluations of cells the kernel modifies the config, appending the components from the newly evaluated cells to config.
• When the modifications are done, the kernel sends SIGHUP to Vector to request reloading of the config.
• After that it reads new output from the automatically created sink and sends it to the user.

If we want, we can write the kernel in Rust instead, but I think that prototyping it in Python would be easier, while we don't need high performance from it and it is supposed to be accompanied by Jupyter which is already written in Python in any case.

Use cases

Using a notebook can make it easier to develop new configs in an interactive manner. And after the experiments are done, converting a notebook to a production-ready Vector config is a matter of taking the content of TOML cells and putting them all together into a config file, which can even be done automatically if necessary.

Start by reading the Jupyter wrapper-kernels documentation and the linked Vector notebook example. Define how the external Python kernel should create temporary Vector configs, append components, reload Vector with SIGHUP, and display source or transform output. Done means the kernel works with Jupyter Notebook, Jupyter Lab, and IPython for the stated component and replacement workflows.