Memory usage grows indefinitely when setting same key within eviction interval
- Lenguaje dominante
- Go
- Estrellas
- 8.2k
- Forks
- 614
- Merge medio
- 5 d 12 h
- PR fusionados (30 d)
- 1
Descripción
This is effectively a follow-up on #109 which is closed for some reason.
We have experienced a bug in production service when hard limits were removed and OOM killed the app. The app has to hold some data in cache which is then repeatedly re-read from DB and re-set in a fixed interval of ~30 min. Keys for saving data in memory are always the same. What we observed is that after few hours memory consumption in our service has grown above any limit.
I slightly modified code snipped from #109 to experiment and reproduce Bigcache behavior:
```go
package main
import (
"strconv"
"time"
"github.com/allegro/bigcache/v3"
)
func main() {
evictionInteval := time.Minute
cacheCfg := bigcache.DefaultConfig(evictionInteval)
// cacheCfg.CleanWindow = time.Second
cacheCfg.Verbose = false
// cacheCfg.HardMaxCacheSize = 100
cache, _ := bigcache.NewBigCache(cacheCfg)
data := []byte("TESTDATATESTDATATESTDATATESTDATATESTDATATESTDATATESTDATA")
for {
for i := 0; i < 10000; i++ {
if err := cache.Set(strconv.Itoa(i), data); err != nil {
panic(err)
}
}
time.Sleep(100 * time.Millisecond)
}
}
```
The memory usage growth depends __only__ on `evictionInterval`.
So, for instance, on my linux machine, when `evictionInterval` set to
- 1 minute, RSS is ~877M
- 2 minutes, RSS is ~1680M
- etc
So if `evictionInterval` is big enough and we keep setting data with same _key_, we would end up with OOM killer.
It doesn't matter whether `GODEBUG=madvdontneed=1` is set or not. I run with this param, but it just seems to not affect anything.
Please note commented `// cacheCfg.CleanWindow = time.Second` line - I tried setting this param to different values starting from 1 sec, and it didn't help.
As a result the only way to limit memory consumption and prevent OOM is to set `HardMaxCacheSize`.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.