Codec's #register method is invoked a second time with the actual params and not the original cloned
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
**Logstash information**:
Please include the following information:
1. Logstash version (e.g. `bin/logstash --version`)
2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
**Plugins installed**: (`bin/logstash-plugin list --verbose`)
**JVM** (e.g. `java -version`):
If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:
1. JVM version (`java -version`)
2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
3. Value of the `LS_JAVA_HOME` environment variable if set.
**OS version** (`uname -a` if on a Unix-like system):
**Description of the problem including expected versus actual behavior**:
The `register` method of codecs happens during pipeline startup. That method should be invoked with the original param that the user configured in the pipeline. The register method is invoked twice:
- the first time to instantiate the codec.
- the second due to the a cloning request during the input instantiation.
While the first time it receives the expected `original_params` with the params explicitly configured by the user in the pipeline config, the second one receives all the parameters.
First time call stack sample:
```
vendor/bundle/jruby/3.1.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'
logstash-core/lib/logstash/codecs/base.rb:33:in `initialize'
logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'
logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'
logstash-core/lib/logstash/pipeline_action/create.rb:49:in `new'
logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'
logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'
```
Second time call stack:
```
vendor/bundle/jruby/3.1.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'
logstash-core/lib/logstash/codecs/base.rb:33:in `initialize'
logstash-core/lib/logstash/codecs/base.rb:94:in `initialize_plugin'
logstash-core/lib/logstash/codecs/base.rb:94:in `clone'
logstash-core/lib/logstash/util.rb:224:in `deep_clone'
logstash-core/lib/logstash/util.rb:218:in `block in deep_clone'
logstash-core/lib/logstash/util.rb:218:in `inject'
logstash-core/lib/logstash/util.rb:218:in `deep_clone'
logstash-core/lib/logstash/plugin.rb:77:in `initialize'
logstash-core/lib/logstash/inputs/base.rb:72:in `initialize'
vendor/bundle/jruby/3.1.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/selector.rb:61:in `block in initialize'
vendor/bundle/jruby/3.1.0/gems/logstash-input-stdin-3.4.0/lib/logstash/inputs/stdin.rb:28:in `initialize'
logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'
logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'
```
**Steps to reproduce**:
Please include a *minimal* but *complete* recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.
1. edit a plugin's register to output the parameters, like
```
def register
puts "DBG>> original_params: #{original_params}"
end
```
2. then configure the path to the development plugin in your `Gemfile`
```
gem "logstash-codec-json_lines", :path => "/path/to/logstash_plugins/logstash-codec-json_lines"
```
```sh
bin/logstash-plugin install --no-verify
```
3. Start Logstash with a pipeline that contains that codec in an input
### Additional notes
Very likely the problem is in cloning of codec https://github.com/elastic/logstash/blob/b88e23702c100d411da8ce340d3c6db90dd6c0d9/logstash-core/lib/logstash/codecs/base.rb#L94 where it passes down `params` instead of `original_params`.
This come up also in https://github.com/logstash-plugins/logstash-codec-json_lines/pull/46#discussion_r1745552405
**Provide logs (if relevant)**:
```
[2024-09-05T16:18:19,861][INFO ][org.reflections.Reflections] Reflections took 42 ms to scan 1 urls, producing 139 keys and 484 values
DNADBG>> original_params: {"id"=>"29dc41a3-3a5f-40d9-965f-37d0f097ee81"}
[2024-09-05T16:18:19,896][INFO ][logstash.codecs.jsonlines] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
DNADBG>> original_params: {"id"=>"29dc41a3-3a5f-40d9-965f-37d0f097ee81", "enable_metric"=>true, "charset"=>"UTF-8", "delimiter"=>"\n", "decode_size_limit_bytes"=>536870912}
[2024-09-05T16:18:19,906][INFO ][logstash.codecs.jsonlines] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2024-09-05T16:18:19,946][INFO ][logstash.javapipeline ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2024-09-05T16:18:19,956][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, "pipeline.sources"=>["/Users/andrea/workspace/logstash_plugins/logstash-codec-json_lines/test_oom_pipeline.conf"], :thread=>"#"}
[2024-09-05T16:18:20,168][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.21}
[2024-09-05T16:18:20,177][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2024-09-05T16:18:20,184][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in logstash-core/lib/logstash/codecs/base.rb at the clone and initialize_plugin logic around line 94, then trace logstash-core/lib/logstash/util.rb deep_clone. Reproduce with the logstash-codec-json_lines development plugin and the minimal pipeline described in the issue. Done means the codec's second register call receives the original user-configured parameters rather than the expanded parameter set.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, ruby
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100