bencheeorg / bencheeorg/benchee

Custom collectors

Open
#444 7 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Elixir
Stars
1.5k
Forks
70
PR merge metrics
No merged PRs in 30d

Description

edit by @PragTob: This is the original issue, we arrived at the solution though that custom collectors would solve this: https://github.com/bencheeorg/benchee/issues/444#issuecomment-2573561240

Hello,

In order to not pollute the computation of something with the process dictionary being already fed with memoized results, I have a function that I would benchmark which boils down to basically this:

```elixir
fn input ->
parent = self()
spawn_link(fn -> send(parent, acutal_computation(input)) end)
receive(do: (msg -> msg))
end
```

Problem is that the "entrypoint" that I can give to benchee must be the external function, to keep the isolation.

Is there a way to provide benchee with a special scenario, something like that:

```elixir
fn input ->
parent = self()
spawn_link(fn ->
_result = acutal_computation(input) # not used for benchmark
measurements = Benchee.some_function_to_capture_measurements()
send(parent, measurements)
end)
# return the measurements to benchee
receive(do: (msg -> msg))
end
```

I want to do that first because the measurements should be taken deeper in the call stack, here it's just for the example. And secondly because for memory consumption ~I don't know if benchee looks at the whole system or just the current process (I guess it does the segond given the code I have read)~. So If the function spawn the memory will be off.

Edit:

> Like memory measurements, this only tracks reductions directly in the function given to benchee, not processes spawned by it or other processes it uses.

Thank you

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked issue comment and the custom collectors approach described in the issue, then inspect Benchee's existing measurement and benchmark entry points. Define how measurements captured inside spawned processes are returned to the benchmark and how completion is verified for memory and reduction metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.