intuit / intuit/go-loadgen

Generation is very slow in `replay-file` mode

Open
#4 10 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
14
Forks
6
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
We are trying to generate logs using a predefined pattern and use `replay-file` for this. But looking into the output, we see that generation rate is very slow ~ 600 line per second.

**To Reproduce**

in.log
```
209.124.125.232 - franecki2473 [22/Dec/2022:18:56:54 +0000] "GET /value-added/repurpose/partnerships/cultivate HTTP/2.0" 403 5187
```

```bash
loadgen replay-file \
--duration 5 \
--input-file-path in.log \
--output out.log \
--result-log result.log \
--lines-per-second 20000 \
--enable-metrics
```

**Expected behavior**
Application should be able to generate logs with a higher speed

**Desktop:**
- OS: m5a.large / Ubuntu 22.04

**Additional context**
```bash
while true; do
processed=$(curl http://localhost:8080/metrics -s | awk '/^total_events_processed/ {print $2}')
[[ -z ${precedent} ]] && precedent=0
rate=$((processed-precedent))
echo "`date` - ${processed} - ${rate}"
sleep 1
precedent=${processed}
done
```
```
Thu Dec 22 20:33:18 UTC 2022 - 353 - 353
Thu Dec 22 20:33:19 UTC 2022 - 1639 - 1286
Thu Dec 22 20:33:20 UTC 2022 - 2924 - 1285
Thu Dec 22 20:33:21 UTC 2022 - 4221 - 1297
Thu Dec 22 20:33:22 UTC 2022 - 5497 - 1276
Thu Dec 22 20:33:23 UTC 2022 - 6799 - 1302
Thu Dec 22 20:33:24 UTC 2022 - 8083 - 1284
Thu Dec 22 20:33:25 UTC 2022 - 9385 - 1302
Thu Dec 22 20:33:26 UTC 2022 - 10671 - 1286
Thu Dec 22 20:33:27 UTC 2022 - 11955 - 1284
Thu Dec 22 20:33:28 UTC 2022 - 13251 - 1296
Thu Dec 22 20:33:29 UTC 2022 - 14551 - 1300
Thu Dec 22 20:33:30 UTC 2022 - 15847 - 1296
Thu Dec 22 20:33:31 UTC 2022 - 17133 - 1286
Thu Dec 22 20:33:32 UTC 2022 - 18427 - 1294
```

In the metrics we see that rate is ~ 1200 line per seconds, but if we look into the file
```bash
cat out.log | grep 2022-12-22T20:33:33 | wc -l
634
```
we can see just half of the number compared to the metric one.

Contributor guide

Open the contributing guide

Research direction

Start with the replay-file mode using the exact loadgen command and input shown, then compare total_events_processed with the lines written to out.log. Investigate why the observed generation rate is far below --lines-per-second and why metrics exceed the matching output lines; done means substantially faster generation with consistent metrics and output counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.