letsencrypt / letsencrypt/boulder

`max_statement_time` is ineffective in multiple-query scenarios

Open
#3,463 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/sa kind/performance
Dominant language
Go
Stars
5.8k
Forks
649
Avg merge
3d 23h
Merged PRs (30d)
24

Description

In sa/database.go, we set the SQL connection-level variable max_statement_time to tell the MySQL server to terminate queries if they are going to take longer than a configured timeout. However, this assumes that every query can take the full allotted time. In some situations, notably RA.NewCertificate, we may be making multiple RPC calls to the SA, and with each call there is less time left available before the RPC deadline. Ideally we would set max_statement_time on a per-query level based on the amount of time left to that query. However, we used to do this, and it triggered a hard-to-diagnose memory leak in MariaDB: https://jira.mariadb.org/browse/MDEV-12205.

To some extent we don't care that much whether we get a max_statement_time exceeded error vs a deadline exceeded error. However, the same code is also used to set a long_query_time, so fixing it might allow us to see the query that caused slowness in some cases.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in sa/database.go and trace how max_statement_time and long_query_time are applied during the multiple SA calls made by RA.NewCertificate. Review the MariaDB MDEV-12205 context before choosing a per-query approach. Done means remaining RPC time is respected without reintroducing the reported memory leak, while preserving useful long-query reporting.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, mariadb, mysql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.