jackc / jackc/pgx

Cache expensive reflection calls

Open
#2,401 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
14.3k
Forks
1.1k
Avg merge
6d 9h
Merged PRs (30d)
11

Description

**Is your feature request related to a problem? Please describe.**
I'd like to use `pgx.RowToStructByName` instead of scany, but currently, it regenerates the struct scan definition on every call. Since the number of structs in a program is limited, we should cache these definitions.

**Describe the solution you'd like**
Cache the fields for each type in the following code to avoid generating them every time.
https://github.com/jackc/pgx/blob/a2fca037434a0a7096b095d4ed87cdffb03b626e/rows.go#L685-L694

**Describe alternatives you've considered**
Keep using scany

**Additional context**

Refs:
- https://github.com/stackrox/stackrox/pull/17126

The patch to change from scany to pgx with a sample bench test

```diff
diff --git a/pkg/search/postgres/select.go b/pkg/search/postgres/select.go
index 6b25478c04..4cd05f66ec 100644
--- a/pkg/search/postgres/select.go
+++ b/pkg/search/postgres/select.go
@@ -7,8 +7,7 @@ import (
"strings"
"time"

- "github.com/georgysavva/scany/v2/dbscan"
- "github.com/georgysavva/scany/v2/pgxscan"
+ "github.com/jackc/pgx/v5"
"github.com/pkg/errors"
v1 "github.com/stackrox/rox/generated/api/v1"
"github.com/stackrox/rox/pkg/env"
@@ -19,27 +18,8 @@ import (
"github.com/stackrox/rox/pkg/search/paginated"
"github.com/stackrox/rox/pkg/search/postgres/aggregatefunc"
pgsearch "github.com/stackrox/rox/pkg/search/postgres/query"
- "github.com/stackrox/rox/pkg/utils"
)

-var scanAPI = newScanAPI(newDBScanAPI(dbscan.WithAllowUnknownColumns(true)))
-
-func newScanAPI(dbscanAPI *dbscan.API) *pgxscan.API {
- api, err := pgxscan.NewAPI(dbscanAPI)
- if err != nil {
- utils.Must(err)
- }
- return api
-}
-
-func newDBScanAPI(opts ...dbscan.APIOption) *dbscan.API {
- api, err := pgxscan.NewDBScanAPI(opts...)
- if err != nil {
- utils.Must(err)
- }
- return api
-}
-
// RunSelectRequestForSchema executes a select request against the database for given schema. The input query must
// explicitly specify select fields.
// Deprecated: Use RunSelectRequestForSchemaFn
@@ -160,10 +140,9 @@ func retryableRunSelectRequestForSchemaFn[T any](ctx context.Context, db postgre
}
defer rows.Close()

- scanner := scanAPI.NewRowScanner(rows)
for rows.Next() {
- var row T
- if err := scanner.Scan(&row); err != nil {
+ row, err := pgx.RowToStructByName[T](rows)
+ if err != nil {
return err
}
if err := fn(&row); err != nil {
```

```
│ scany.txt │ pgx.txt │
│ max_heap_bytes │ max_heap_bytes vs base │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8 11.58M ± 8% 12.64M ± 55% +9.16% (p=0.015 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8 14.37M ± 24% 14.45M ± 7% ~ (p=0.937 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8 15.94M ± 24% 16.31M ± 24% ~ (p=0.485 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8 21.57M ± 23% 21.54M ± 19% ~ (p=0.937 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8 19.93M ± 17% 21.26M ± 3% ~ (p=0.394 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8 31.26M ± 25% 42.44M ± 9% +35.75% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8 41.02M ± 35% 53.73M ± 12% ~ (p=0.132 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8 38.95M ± 46% 67.87M ± 34% +74.25% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8 80.47M ± 17% 92.02M ± 11% ~ (p=0.093 n=6)
geomean 25.52M 29.96M +17.41%

│ master.txt │ new.txt │
│ max_heap_objects │ max_heap_objects vs base │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8 11.58M ± 8% 12.64M ± 55% +9.16% (p=0.015 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8 14.37M ± 24% 14.45M ± 7% ~ (p=0.937 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8 15.94M ± 24% 16.31M ± 24% ~ (p=0.485 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8 21.57M ± 23% 21.54M ± 19% ~ (p=0.937 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8 19.93M ± 17% 21.26M ± 3% ~ (p=0.394 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8 31.26M ± 25% 42.44M ± 9% +35.75% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8 41.02M ± 35% 53.73M ± 12% ~ (p=0.132 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8 38.95M ± 46% 67.87M ± 34% +74.25% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8 80.47M ± 17% 92.02M ± 11% ~ (p=0.093 n=6)
geomean 25.52M 29.96M +17.41%

│ master.txt │ new.txt │
│ B/op │ B/op vs base │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8 660.8Ki ± 0% 938.6Ki ± 0% +42.04% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8 2.031Mi ± 0% 3.400Mi ± 0% +67.38% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8 3.616Mi ± 0% 6.354Mi ± 0% +75.73% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8 8.794Mi ± 0% 11.536Mi ± 0% +31.19% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8 6.680Mi ± 0% 12.168Mi ± 0% +82.14% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8 31.70Mi ± 0% 59.16Mi ± 0% +86.61% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8 77.96Mi ± 0% 146.62Mi ± 0% +88.07% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8 155.1Mi ± 0% 292.4Mi ± 0% +88.57% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8 442.4Mi ± 0% 579.7Mi ± 0% +31.02% (p=0.002 n=6)
geomean 15.35Mi 25.19Mi +64.16%

│ master.txt │ new.txt │
│ allocs/op │ allocs/op vs base │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8 15.45k ± 0% 16.39k ± 0% +6.09% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8 57.83k ± 0% 62.76k ± 0% +8.53% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8 108.5k ± 0% 118.4k ± 0% +9.12% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8 108.5k ± 0% 118.5k ± 0% +9.16% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8 208.5k ± 0% 228.4k ± 0% +9.56% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8 1.015M ± 0% 1.115M ± 0% +9.85% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8 2.515M ± 0% 2.765M ± 0% +9.94% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8 5.015M ± 0% 5.515M ± 0% +9.97% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8 5.016M ± 0% 5.516M ± 0% +9.96% (p=0.002 n=6)
geomean 373.3k 407.4k +9.12%
```

Contributor guide

Open the contributing guide

Research direction

Start in rows.go at the referenced RowToStructByName code around lines 685-694, and trace how the struct scan definition is generated for each call. The work is complete when definitions are reused per type without changing row-to-struct behavior; use the issue's migration benchmark context to assess whether repeated reflection and allocations improve.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.