influxdata / influxdata/influxdb-client-ruby
Memory leak
- Dominant language
- Ruby
- Stars
- 50
- Forks
- 22
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
We spotted what seems to be a memory leak in the gem, when sending lots of data to InfluxDB.
## Steps to reproduce
1. Leave the code running in production, sending lots of data through the gem
2. Observe that memory usage keeps increasing
This is about the code that we use to setup the gem:
```ruby
module Influxdb
CLIENT = InfluxDB2::Client.new(url, token, precision: InfluxDB2::WritePrecision::MILLISECOND)
WRITE_OPTIONS = InfluxDB2::WriteOptions.new(
write_type: InfluxDB2::WriteType::BATCHING,
batch_size: 10,
flush_interval: 5_000,
max_retries: 25,
retry_interval: 1_000,
max_retry_delay: 30_000,
max_retry_time: 300_000,
)
WRITE_API = CLIENT.create_write_api(write_options: WRITE_OPTIONS)
# @param [String] bucket
# @param [String] name
# @param [Hash] values
# @param [Hash] tags
def self.write(bucket:, name:, tags: {}, values:)
WRITE_API.write(
org: "my-org",
bucket:,
data: {
name: name,
tags: tags,
fields: values,
time: Time.zone.now,
},
)
end
end
```
Then data is sent with calls to `Influxdb.write`, like so:
```ruby
Influxdb.write(bucket: "some-bucket", name: "some-operation", values: { duration: }, tags: { extra: "info" })
```
## Test locally
We ran some tests with out regular rails app:
```bash
rails console
```
```ruby
# before doing anything, our rails app consumes 250MB
(1..1_000_000).each do |i|
puts i
Influxdb.write(bucket: "test", name: "blah", values: {duration: rand(1000)})
end
## RAM slowly increases to 700MB RAM
```
### Note on `batch_size`
Increasing `batch_size` to `1_000` as suggested in the defaults has a drastic impact.
Memory usage is increasing much slower with `batch_size: 1_000` instead of `10`, but it does keep increasing.
## Expected behavior
We would expect the RAM to be released once the data is sent.
## Actual behavior
RAM is never released.
Do we have to close the client regularly, or something similar, so that it flushes the memory?
## Specifications
- Client Version: `gem "influxdb-client", "~> 3.2"`
- InfluxDB Version: `InfluxDB v2.7.12`
- Platform: `Ubuntu 22.04 LTS`
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the report using InfluxDB2::Client.new, create_write_api, and repeated WRITE_API.write calls with the shown batching options. Measure memory during sustained writes and investigate whether memory is released after data is sent or whether the client must be closed; done means the leak is reproduced, its cause is identified, and behavior is covered by an appropriate regression check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100