elastic / elastic/elastic-package

benchmarking: mapping for processors is wrong

Open
#2,498 0 comments 0 reactions 0 assignees View on GitHub
Team:Ecosystem
Dominant language
Go
Stars
72
Forks
141
Avg merge
19h 42m
Merged PRs (30d)
55

Description

When running benchmarks, the mapping between the processors listed in the output and the actual source appears to be broken.

```
$ git rev-parse HEAD
1595cc1388bf5054c47d5828535ff772944804df
$ pwd
…/github.com/elastic/integrations/packages/sentinel_one_cloud_funnel
$ elastic-package benchmark pipeline -d event

Run pipeline benchmarks for the package
--- Benchmark results for package: sentinel_one_cloud_funnel - START ---
╭─────────────────────────╮
│ parameters │
├──────────────────┬──────┤
│ source_doc_count │ 16 │
│ doc_count │ 1000 │
╰──────────────────┴──────╯
╭───────────────────────────╮
│ pipeline_performance │
├─────────────────┬─────────┤
│ processing_time │ 0.69s │
│ eps │ 1440.92 │
╰─────────────────┴─────────╯
╭────────────────────────────────────────╮
│ procs_by_total_time │
├───────────────────────────────┬────────┤
│ dot_expander @ default.yml:52 │ 19.31% │
│ json @ default.yml:34 │ 4.61% │
│ remove @ default.yml:2778 │ 3.89% │
│ script @ default.yml:2849 │ 1.73% │
│ date @ default.yml:87 │ 1.44% │
│ date @ default.yml:533 │ 1.44% │
│ date @ default.yml:1723 │ 1.30% │
│ gsub @ default.yml:20 │ 1.01% │
│ script @ default.yml:2770 │ 1.01% │
│ script @ pipeline-dns.yml:21 │ 0.72% │
╰───────────────────────────────┴────────╯
╭──────────────────────────────────────────╮
│ procs_by_avg_time_per_doc │
├───────────────────────────────┬──────────┤
│ dot_expander @ default.yml:52 │ 134µs │
│ json @ default.yml:34 │ 32µs │
│ remove @ default.yml:2778 │ 27µs │
│ script @ pipeline-dns.yml:21 │ 19.841µs │
│ script @ default.yml:2849 │ 12µs │
│ date @ default.yml:533 │ 10.66µs │
│ date @ default.yml:87 │ 10µs │
│ date @ default.yml:1723 │ 9µs │
│ script @ default.yml:2770 │ 7.462µs │
│ gsub @ default.yml:20 │ 7µs │
╰───────────────────────────────┴──────────╯

--- Benchmark results for package: sentinel_one_cloud_funnel - END ---
Done
```

But to take one example, the `dot_expander` since it's unique and near the head of the file, it is not at line 52, it's at [44](https://github.com/elastic/integrations/blob/1595cc1388bf5054c47d5828535ff772944804df/packages/sentinel_one_cloud_funnel/data_stream/event/elasticsearch/ingest_pipeline/default.yml#L44).

Internal discussion has raised the possibility that this is due to the presence of reroute rules in the package, something that was found to be an issue in the pipeline test runner (ref:#1647).

Another thing that would help here is to not only identify the processor by its line number, but to also use its tag if it exists.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `elastic-package benchmark pipeline -d event` entry point and reproduce the output against `packages/sentinel_one_cloud_funnel/data_stream/event/elasticsearch/ingest_pipeline/default.yml`. Compare reported processor lines with the processors in `default.yml`, including the `dot_expander` location, and investigate whether reroute rules cause the offset described in ref:#1647. Done means processor mappings are accurate and processor tags are shown when available.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.