influxdata / influxdata/influxdb-client-ruby

Memory leak

Open
#147 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
50
Forks
22
PR merge metrics
No merged PRs in 30d

Description

## Summary

We spotted what seems to be a memory leak in the gem, when sending lots of data to InfluxDB.

## Steps to reproduce

1. Leave the code running in production, sending lots of data through the gem
2. Observe that memory usage keeps increasing

This is about the code that we use to setup the gem:

```ruby
module Influxdb
CLIENT = InfluxDB2::Client.new(url, token, precision: InfluxDB2::WritePrecision::MILLISECOND)

WRITE_OPTIONS = InfluxDB2::WriteOptions.new(
write_type: InfluxDB2::WriteType::BATCHING,
batch_size: 10,
flush_interval: 5_000,

max_retries: 25,
retry_interval: 1_000,
max_retry_delay: 30_000,
max_retry_time: 300_000,
)

WRITE_API = CLIENT.create_write_api(write_options: WRITE_OPTIONS)

# @param [String] bucket
# @param [String] name
# @param [Hash] values
# @param [Hash] tags
def self.write(bucket:, name:, tags: {}, values:)
WRITE_API.write(
org: "my-org",
bucket:,
data: {
name: name,
tags: tags,
fields: values,
time: Time.zone.now,
},
)
end
end
```

Then data is sent with calls to `Influxdb.write`, like so:
```ruby
Influxdb.write(bucket: "some-bucket", name: "some-operation", values: { duration: }, tags: { extra: "info" })
```

## Test locally

We ran some tests with out regular rails app:

```bash
rails console
```
```ruby
# before doing anything, our rails app consumes 250MB

(1..1_000_000).each do |i|
puts i
Influxdb.write(bucket: "test", name: "blah", values: {duration: rand(1000)})
end

## RAM slowly increases to 700MB RAM
```

### Note on `batch_size`

Increasing `batch_size` to `1_000` as suggested in the defaults has a drastic impact.

Memory usage is increasing much slower with `batch_size: 1_000` instead of `10`, but it does keep increasing.

## Expected behavior

We would expect the RAM to be released once the data is sent.

## Actual behavior

RAM is never released.

Do we have to close the client regularly, or something similar, so that it flushes the memory?

## Specifications
- Client Version: `gem "influxdb-client", "~> 3.2"`
- InfluxDB Version: `InfluxDB v2.7.12`
- Platform: `Ubuntu 22.04 LTS`

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the report using InfluxDB2::Client.new, create_write_api, and repeated WRITE_API.write calls with the shown batching options. Measure memory during sustained writes and investigate whether memory is released after data is sent or whether the client must be closed; done means the leak is reproduced, its cause is identified, and behavior is covered by an appropriate regression check.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.