elastic / elastic/logstash

gems (and underlying net/http) is severly slow on Jruby

Open
#4,971 5 comments 0 reactions 1 assignee Claimed by @purbon View on GitHub
enhancement
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

After falsely suspecting octokit, I found out that the **gems** gem and in fact any **repeated calls to net/http** speed are heavily slow under jruby.

As an example I did a micro-benchmark on the method [is_released? ](https://github.com/elastic/logstash/blob/master/rakelib/default_plugins.rb#L161-L164) looping on the list of DEFAULT_PLUGINS

```
require 'gems'
require 'faraday'
require 'manticore'
require 'benchmark'
Benchmark.bm do |x|
x.report("current") { DEFAULT_PLUGINS.each { |p| Gems.info(p) != "This rubygem could not be found."} }
x.report("faraday") { DEFAULT_PLUGINS.each { |p| Faraday.get("https://rubygems.org/api/v1/versions/#{p}/latest.json").body != '{"version":"unknown"}'} }
x.report("manticore") { DEFAULT_PLUGINS.each { |p| Manticore.get( "https://rubygems.org/api/v1/versions/#{p}/latest.json").body != '{"version":"unknown"}'} }
end
```

And the result is clear, and

```
user system total real
current 9.190000 0.160000 9.350000 (114.753000)
faraday 5.900000 0.140000 6.040000 (112.954000)
manticore 1.250000 0.040000 1.290000 ( 23.931000)
```

I suppose it can be an improvment to #4746 and this could also impact the plugin manager where list of plugins are processed

Can a logstash ruby/jruby expert confirm these points:
- there is no jruby net/http config that can make it equivalent in speed to manticore ?
- should we invest time in refactoring logstash usage of **gems** and use an internal custom implementation based on rubygem JSON API and manticore http client ?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.