[RFC] Plugin Redesign

This RFC proposes a new approach for supporting plugins with Data Prepper. Its main goals are to promote modularity and allow for decentralized plugins.

What is the problem? What is preventing you from meeting the requirements?
Data Prepper plugins should be split out so that they do not need to be part of every installation and Data Prepper. Additionally, other teams or individuals should be able to easily create their own plugins for Data Prepper. These remote plugins would be development and deployed independently of Data Prepper.

Currently, Data Prepper requires that all the plugins be embedded within the jar and in the current classpath.

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?

Data Prepper will support loading plugins from two different sources:

• The Java classpath of Data Prepper
• External plugins from remote repositories

This diagram outlines the division between the two types of plugins.

The Java classpath will be used for two scenarios:

1. Any core plugin which will always be included within Data Prepper
2. Custom Data Prepper distributions. Users of Data Prepper may be running Data Prepper without internet access (e.g. running in an enclosed network), and should be able to install custom versions of Data Prepper with all the plugins they need.

Loading plugins from remote repositories will be used for plugins which are not currently installed in Data Prepper. This proposal uses Maven Central as the mechanism for distribution of these plugins. Additionally, plugins will load within their own dedicated class loader to provide isolation between plugins. A future RFC or proposal will detail this approach.

This diagram shows the concept for supporting remote plugins which are external to Data Prepper.

Remote plugins will be downloaded into their own plugins directory within the Data Prepper directory structure.

The following outlines a possible directory structure. In this approach, each plugin has its own uber-jar file with all of its dependencies include. It may also be worth considering expanding a plugin uber jar as an alternative.

data-prepper-$VERSION/
  bin/
    data-prepper                    # Shell script to run Data Prepper on Linux/macOS
  plugins/
    my-plugin-a.jar
    my-plugin-b.jar
  logs/                             # Directory for log output
  LICENSE
  NOTICE
  README.md

What are your assumptions or prerequisites?

This work depends on the new directory structure being added as part of Directory Structure for Data Prepper.

Tasks

• #1543
• Support loading plugins from a remote repository

Start by reviewing the prerequisite Directory Structure for Data Prepper issue (#305) and task #1543. Then assess the proposed classpath and remote-repository plugin sources, dedicated class loaders, and plugins directory; the work is done when the agreed redesign is implemented and its support boundaries are documented.