Enable Dynamic Thread Pool to Improve Memory Usage
- Dominant language
- Ruby
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 20
Description
During our benchmark tests, we found that the memory usage of Fluentd increases as num_threads increases (we keep the other configurations exactly the same). It's almost linear.

We are testing with pretty low QPS, which should not require that many threads as specified in `num_threads`. It makes sense to pre-allocate memory for each thread. But seems Fluentd could have saved a lot of resource if it uses some dynamic thread pool / kill idle threads and add new ones only when necessary. That way the baseline of memory usage should be lower.
Is this something that's possible to enable with the current Fluentd (we haven't tried `v1.0.X` yet)?
Thanks!
- Fluentd version
`v0.14.25`
- OS
`ubuntu-1604-xenial`
- Configuration
```
@type forward
port 24224
bind 127.0.0.1
@type prometheus
port 24231
@type prometheus_monitor
@type google_cloud
buffer_chunk_limit 1M
flush_interval 5s
max_retry_wait 300
disable_retry_limit
num_threads 8
use_grpc true
enable_monitoring true
monitoring_type prometheus
```
Contributor guide
Assessment
This issue has not been assessed yet.