elastic / elastic/beats

Metricbeat sql module - MySql connection leak when query uses prepared statements

Open
#44,335 3 comments 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

Discuss:
https://discuss.elastic.co/t/metricbeat-8-sql-module-leaks-connections-when-using-prepared-statements/377684

Metricbeat Version 8.14.3
Amazon Linux 2 w/ MySql 8.0.28, Also AWS Aurora Mysql 8.0.36

We recently went from Metricbeat 7.14 to 8.14
We have found idle metricbeat connections piling up in mysql.

Problem: We use the sql module to run queries on mysql. The queries use prepared statements. Since going to metricbeat 8, the connections leak, maybe a difference in the GOlang driver. The queries still work but the connections stay until we hit the mysql connection limit.

Steps to duplicate: spin up a mysql 8.0 instance and use this sql.yml to query it.

Set up /etc/metricbeat/modules.d/sql.yaml to query mysql with a prepared statement:

```
# Module: sql
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/8.14/metricbeat-module-sql.html

- module: sql
metricsets:
- query
period: 10s
hosts: ["metricbeat:password@tcp(127.0.0.1:3306)/?multiStatements=true"]
driver: "mysql"
sql_query: |
SET @schema_name = 'mysql';
SET @sql = CONCAT(
'SELECT TABLE_NAME, TABLE_TYPE ',
'FROM information_schema.tables ',
'WHERE TABLE_SCHEMA = \'', @schema_name, '\''
);
PREPARE stmt FROM @sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
sql_response_format: table
```

Check connections to mysql

![Image](https://github.com/user-attachments/assets/8316f3f6-78a1-4d01-9aea-52c8ec58b3a4)

These queries work fine, but I have set up a separate job to kill the connections because they increase until we hit the connection limit. Seems like a drastic step.
I have tried to convert my queries into stored procedures, and then used metricbeat to call the proc. Unfortunately same result.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the leak with the provided sql.yml against MySQL 8.0, using the prepared-statement query and checking idle connections. Then trace the Metricbeat SQL module's connection lifecycle and verify that repeated queries, stored procedures, and prepared statements release connections without reaching the MySQL limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, mysql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.