Imprecise counters to avoid issues with contended `atomicModifyIORef`
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- haskell
- Domain
- observability-sre, performance
Research direction
No source file, test, or entry point is named. Start by locating the EKG counter implementation and its UI representation, then compare the existing atomicModifyIORef approach with the proposed per-capability byte-array design. Done means counters use the imprecise representation, values are summed for reads, and the UI identifies them as imprecise.
Written by the indexing model from the issue text.
Description
[Since @tibbe wanted to be kept in the loop.] We're currently testing if these ideas are ready for our production system and then will integrate them back into upstream. So, here's a quick summary of what we're doing:
Problem: We use a lot of EKG counters to keep track of various code paths of our system. Each counter is incremented using atomicModifyIORef and our binaries usually run with 4 or more hardware threads. We noticed on microbenchmarks that these counters add a non-trivial overhead, most likely because of lots of cache-line contention an the busy-waiting loop used by atomicModifyIORef. We could use modifyIORef, but that would likely get less and less precise the more cores are involved.
Proposed Solution: The basic idea is to use one counter per capability and then sum them up when we reading the current value. This is still a bit imprecise, but probably much less so than the IORef approach. So, you represent a counter as a byte array and each capability writes to a different part of that byte array. These writes don't have to be atomic. For a slight improvement in performance we also make sure that two cores don't share the same cache line. The other parts of the cache line can be used for other counters. So the array looks as follows:
| capability 0 | capabality 1 | capability 2 |
+-----------------------------------------------------------+
| c1 | c2 | c3 | c4 | c1 | c2 | c3 | c4 | c1 | c2 | c3 | c4 |
+-----------------------------------------------------------+
|<---- stride ----->|
The per-capability counters for counter c1 are at offset 0, 1 * stride + 0, 2 * stride + 0. The current value of the counter is the sum of all of these per-capability counters. Since we cannot read all values at the same time, we have a race condition, but that's why they're imprecise counters.
Due to this imprecision they should be displayed as imprecise counters in the UI as well.
- Dominant language
- JavaScript
- Stars
- 255
- Forks
- 66
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from haskell-github-trust/ekg
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
haskell-github-trust/ekg#97 · 1 comment ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
haskell-github-trust/ekg#87 · 1 comment ·
-
Add tags support Open
Difficulty 5/5 Over a week Newbie friendliness 20/100
haskell-github-trust/ekg#75 · 1 comment ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
haskell-github-trust/ekg#73 · 1 comment ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
haskell-github-trust/ekg#71 ·
All issues in haskell-github-trust/ekg
Similar issues
-
code-quality refactoring
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
github/gh-aw-firewall#8816 ·
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
optimization optimization:agents-md-curator
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
githubnext/gh-aw-cao#13143 ·
-
status: needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100