algorand / algorand/go-algorand

Degradation of warm up time for algod docker image since 3.16.X

Open
#5,498 1 comment 1 reaction 1 assignee Claimed by @shiqizng View on GitHub
bug Team Lamprey
Dominant language
Go
Stars
1.4k
Forks
537
Avg merge
1d 6h
Merged PRs (30d)
17

Description

### Subject of the issue

The warm up time (that is the time it takes for endpoints to start responding successfully after the container is first started) for the official algod docker images when using `START_KMD=1` has significantly increased between versions `3.15.1` and `3.16.x` (tested on `3.16.0-beta` and `3.16.2-stable`)

*Observation:* There is a 30s delay between the following two algod log messages, which may imply kmd is taking 30s to start
```
2023-06-26 14:13:15 Telemetry logging disabled: Name = , Guid = a10be9a2-6e22-421a-8f03-f2686fec4a29
2023-06-26 14:13:45 Successfully started kmd
```

### Your environment

* Software version: First replicated in `3.16.0-beta`, still present in `3.16.2-stable`
* Operating System: Replicated on multiple OS's, exact times differ but the increase in warm up seems to be consistently around 30s

### Steps to reproduce

The following script can be used to see the difference in start up times, `docker` and `curl` are assumed to be available.
```bash
#!/bin/bash
export TOKEN=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

wait_for_status() {
echo -n "Waiting for /v2/status "
for i in {1..60}; do
curl --fail http://localhost:8080/v2/status --header "X-Algo-API-Token: $TOKEN" &>/dev/null
if [ $? -eq 0 ]; then
echo
echo "Took ${i}s"
return
fi
echo -n .
sleep 1
done
}

get_warmup() {
version=$1
echo "Starting algod:$version"
docker run \
--name algod_warm \
-d \
--env START_KMD=1 \
--env TOKEN=$TOKEN \
-p 8080:8080 \
algorand/algod:$version >/dev/null

wait_for_status
echo "Cleaning up ..."
docker stop algod_warm >/dev/null
docker rm algod_warm >/dev/null
echo
}

get_warmup 3.15.1-stable
get_warmup 3.16.2-stable
```

### Expected behaviour
Warm up time of `3.16.x` is comparable to warm up time of `3.15.1`

### Actual behaviour
Warm up time is significantly longer in `3.16.x`

Sample output of above script from a MacBoox Pro M1 running Ventura 13.2
```
Starting algod:3.15.1-stable
Waiting for /v2/status ....
Took 5s
Cleaning up ...

Starting algod:3.16.2-stable
Waiting for /v2/status ..................................
Took 35s
Cleaning up ...
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.