elastic / elastic/logstash

Java API: Design extension loading for new Java plugins

Open
#8,375 0 comments 0 reactions 0 assignees View on GitHub
discuss
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

Parent ticket: https://github.com/elastic/logstash/issues/7986

Logstash runs primarily plugins, and those plugins are currently all ruby. We have a custom plugin loader in Ruby which works fairly well, and now we need one for Java.

## Background / History

Today, we have `config_name` class method to register a plugin:

```ruby
class MyPlugin < Input
config_name "fancy"
end
```

For Logstash to load a plugin, it has to know where to find the code. Today, we have a mapping of plugin names to the plugin classes, and if no mapping is found, it loads a file named by the plugin `lib/logstash/[type]s/name` like `lib/logstash/inputs/stdin`.

## Goals

We need to support runtime loading of custom Logstash plugins that are written in Java.

* Ideally this is implemented in pure Java (no ruby).

## Considering Prior Art

* Lucene uses an SPI-like approach but uses a custom loader instead of using the built-in ServiceLoader. The format of the META-INFO/services/ files are identical to the SPI specification.
* [Elasticsearch does this for plugins](https://www.elastic.co/guide/en/elasticsearch/plugins/current/plugin-authors.html)
* Could we implement Java SPI completely? `java.util.ServiceLoader` etc.
* Annotations + classpath scanning to find classes implementing a given interface

I talked with @rjernst about how ES does plugins, and long term ES may be moving to use an SPI approach similar to Lucene. I'm down with trying out SPI to see with how it works for Logstash plugins.

## Constraints

* Input/filter/output/codec plugins are likely to be created using ObjectFactory from https://github.com/elastic/logstash/pull/8240 because they are instantiated from user pipeline configs.

Contributor guide

Open the contributing guide

Research direction

Start with parent ticket #7986 and the existing Ruby plugin-loader behavior described here, then review the ObjectFactory work in pull request #8240. Compare Java ServiceLoader with the Lucene and Elasticsearch plugin approaches. Done means a decided design for runtime Java plugin loading that covers input, filter, output, and codec instantiation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, ruby
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.