JuliaParallel / JuliaParallel/Dagger.jl

Logging is slow

Open
#621 4 comments 0 reactions 0 assignees View on GitHub
logging performance
Dominant language
Julia
Stars
723
Forks
90
Avg merge
1d 37m
Merged PRs (30d)
9

Description

In a single multithreaded process, logging with `MultiEventLog` brings massive overheads. For 8 threads and 40 graphs ~800 vertices each we observe ~7x application slow down and ~26x increase of lock conflicts. With the overhead this big logging isn't very useful as it considerably alters the application (way more time is spent on logging than on running the actual application)

Without logging
```txt
Pipeline (throughput 1.00 events/s): 40.097314 seconds (62.55 M allocations: 3.252 GiB, 3.20% gc time, 128376 lock conflicts, 0.47% compilation time)
```

With logging

```julia
Dagger.enable_logging!(tasknames = true,
taskfuncnames = true,
taskdeps = true,
taskargs = true,
taskargmoves = true,
taskresult = true,
taskuidtotid = true,
tasktochunk = true)
```

```txt
Pipeline (throughput 0.14 events/s): 290.967248 seconds (233.25 M allocations: 19.686 GiB, 4.86% gc time, 3346075 lock conflicts, 0.00% compilation time)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.