fluent / fluent/fluentd

Enable Dynamic Thread Pool to Improve Memory Usage

Open
#1,797 2 comments 0 reactions 1 assignee Claimed by @repeatedly View on GitHub
enhancement memory v1
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

During our benchmark tests, we found that the memory usage of Fluentd increases as num_threads increases (we keep the other configurations exactly the same). It's almost linear.

![num_threads v s rss](https://user-images.githubusercontent.com/5287526/34180944-72daac22-e4de-11e7-90a7-0e54d6e14521.png)

We are testing with pretty low QPS, which should not require that many threads as specified in `num_threads`. It makes sense to pre-allocate memory for each thread. But seems Fluentd could have saved a lot of resource if it uses some dynamic thread pool / kill idle threads and add new ones only when necessary. That way the baseline of memory usage should be lower.

Is this something that's possible to enable with the current Fluentd (we haven't tried `v1.0.X` yet)?

Thanks!

- Fluentd version
`v0.14.25`
- OS
`ubuntu-1604-xenial`
- Configuration
```

@type forward
port 24224
bind 127.0.0.1

@type prometheus
port 24231

@type prometheus_monitor

@type google_cloud
buffer_chunk_limit 1M
flush_interval 5s
max_retry_wait 300
disable_retry_limit
num_threads 8
use_grpc true
enable_monitoring true
monitoring_type prometheus

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.