cockroachdb / cockroachdb/cockroach
sql: free scanner and parser allocations after large statement
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
After a large statement is scanned and parsed, we hang onto the scanner and parser allocations in https://github.com/cockroachdb/cockroach/blob/3d054f37c7c87f53cb56fac4e5500f0d1130d09a/pkg/sql/parser/parse.go#L111-L119 until the session is closed. This seems like a temporary memory leak, but with connection pooling we might not actually close the session and free this memory for a long time. We should reset the parser after large statements.
Here's a repro, using python to create a large statement:
```
Python 3.10.10 (main, Feb 8 2023, 05:40:53) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
>>> url = "..."
>>> conn = psycopg.connect(url)
>>> c = conn.cursor()
>>> c.execute("CREATE TABLE IF NOT EXISTS t AS SELECT 0 AS a")
>>> sql = "SELECT count(*) FROM t WHERE a IN (" + ",".join(("0" for _ in range(8000000))) + ")"
>>> len(sql) / 1024 / 1024
15.258822441101074
>>> c.execute(sql)
```
Even though this statement is less than 16 MiB, it causes us to hold onto over a GiB of memory in the scanner and parser, and this is not released until the connection closes:

This issue is somewhat similar to https://github.com/cockroachdb/cockroach/issues/47969 and https://github.com/cockroachdb/cockroach/issues/72581 and https://github.com/cockroachdb/cockroach/issues/80497
Jira issue: CRDB-24639
Contributor guide
Research direction
Start in pkg/sql/parser/parse.go at the referenced lines, then use the Python/psycopg reproduction to observe memory retained after the large statement. Trace the scanner and parser lifetimes and verify that their large allocations are released or reset before the session closes, while confirming normal parsing still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100