Cache expensive reflection calls
- Dominant language
- Go
- Stars
- 14.3k
- Forks
- 1.1k
- Avg merge
- 6d 9h
- Merged PRs (30d)
- 11
Description
**Is your feature request related to a problem? Please describe.**
I'd like to use `pgx.RowToStructByName` instead of scany, but currently, it regenerates the struct scan definition on every call. Since the number of structs in a program is limited, we should cache these definitions.
**Describe the solution you'd like**
Cache the fields for each type in the following code to avoid generating them every time.
https://github.com/jackc/pgx/blob/a2fca037434a0a7096b095d4ed87cdffb03b626e/rows.go#L685-L694
**Describe alternatives you've considered**
Keep using scany
**Additional context**
Refs:
- https://github.com/stackrox/stackrox/pull/17126
The patch to change from scany to pgx with a sample bench test
```diff
diff --git a/pkg/search/postgres/select.go b/pkg/search/postgres/select.go
index 6b25478c04..4cd05f66ec 100644
--- a/pkg/search/postgres/select.go
+++ b/pkg/search/postgres/select.go
@@ -7,8 +7,7 @@ import (
"strings"
"time"
- "github.com/georgysavva/scany/v2/dbscan"
- "github.com/georgysavva/scany/v2/pgxscan"
+ "github.com/jackc/pgx/v5"
"github.com/pkg/errors"
v1 "github.com/stackrox/rox/generated/api/v1"
"github.com/stackrox/rox/pkg/env"
@@ -19,27 +18,8 @@ import (
"github.com/stackrox/rox/pkg/search/paginated"
"github.com/stackrox/rox/pkg/search/postgres/aggregatefunc"
pgsearch "github.com/stackrox/rox/pkg/search/postgres/query"
- "github.com/stackrox/rox/pkg/utils"
)
-var scanAPI = newScanAPI(newDBScanAPI(dbscan.WithAllowUnknownColumns(true)))
-
-func newScanAPI(dbscanAPI *dbscan.API) *pgxscan.API {
- api, err := pgxscan.NewAPI(dbscanAPI)
- if err != nil {
- utils.Must(err)
- }
- return api
-}
-
-func newDBScanAPI(opts ...dbscan.APIOption) *dbscan.API {
- api, err := pgxscan.NewDBScanAPI(opts...)
- if err != nil {
- utils.Must(err)
- }
- return api
-}
-
// RunSelectRequestForSchema executes a select request against the database for given schema. The input query must
// explicitly specify select fields.
// Deprecated: Use RunSelectRequestForSchemaFn
@@ -160,10 +140,9 @@ func retryableRunSelectRequestForSchemaFn[T any](ctx context.Context, db postgre
}
defer rows.Close()
- scanner := scanAPI.NewRowScanner(rows)
for rows.Next() {
- var row T
- if err := scanner.Scan(&row); err != nil {
+ row, err := pgx.RowToStructByName[T](rows)
+ if err != nil {
return err
}
if err := fn(&row); err != nil {
```
```
│ scany.txt │ pgx.txt │
│ max_heap_bytes │ max_heap_bytes vs base │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8 11.58M ± 8% 12.64M ± 55% +9.16% (p=0.015 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8 14.37M ± 24% 14.45M ± 7% ~ (p=0.937 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8 15.94M ± 24% 16.31M ± 24% ~ (p=0.485 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8 21.57M ± 23% 21.54M ± 19% ~ (p=0.937 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8 19.93M ± 17% 21.26M ± 3% ~ (p=0.394 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8 31.26M ± 25% 42.44M ± 9% +35.75% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8 41.02M ± 35% 53.73M ± 12% ~ (p=0.132 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8 38.95M ± 46% 67.87M ± 34% +74.25% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8 80.47M ± 17% 92.02M ± 11% ~ (p=0.093 n=6)
geomean 25.52M 29.96M +17.41%
│ master.txt │ new.txt │
│ max_heap_objects │ max_heap_objects vs base │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8 11.58M ± 8% 12.64M ± 55% +9.16% (p=0.015 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8 14.37M ± 24% 14.45M ± 7% ~ (p=0.937 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8 15.94M ± 24% 16.31M ± 24% ~ (p=0.485 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8 21.57M ± 23% 21.54M ± 19% ~ (p=0.937 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8 19.93M ± 17% 21.26M ± 3% ~ (p=0.394 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8 31.26M ± 25% 42.44M ± 9% +35.75% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8 41.02M ± 35% 53.73M ± 12% ~ (p=0.132 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8 38.95M ± 46% 67.87M ± 34% +74.25% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8 80.47M ± 17% 92.02M ± 11% ~ (p=0.093 n=6)
geomean 25.52M 29.96M +17.41%
│ master.txt │ new.txt │
│ B/op │ B/op vs base │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8 660.8Ki ± 0% 938.6Ki ± 0% +42.04% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8 2.031Mi ± 0% 3.400Mi ± 0% +67.38% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8 3.616Mi ± 0% 6.354Mi ± 0% +75.73% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8 8.794Mi ± 0% 11.536Mi ± 0% +31.19% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8 6.680Mi ± 0% 12.168Mi ± 0% +82.14% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8 31.70Mi ± 0% 59.16Mi ± 0% +86.61% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8 77.96Mi ± 0% 146.62Mi ± 0% +88.07% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8 155.1Mi ± 0% 292.4Mi ± 0% +88.57% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8 442.4Mi ± 0% 579.7Mi ± 0% +31.02% (p=0.002 n=6)
geomean 15.35Mi 25.19Mi +64.16%
│ master.txt │ new.txt │
│ allocs/op │ allocs/op vs base │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8 15.45k ± 0% 16.39k ± 0% +6.09% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8 57.83k ± 0% 62.76k ± 0% +8.53% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8 108.5k ± 0% 118.4k ± 0% +9.12% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8 108.5k ± 0% 118.5k ± 0% +9.16% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8 208.5k ± 0% 228.4k ± 0% +9.56% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8 1.015M ± 0% 1.115M ± 0% +9.85% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8 2.515M ± 0% 2.765M ± 0% +9.94% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8 5.015M ± 0% 5.515M ± 0% +9.97% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8 5.016M ± 0% 5.516M ± 0% +9.96% (p=0.002 n=6)
geomean 373.3k 407.4k +9.12%
```
Contributor guide
Research direction
Start in rows.go at the referenced RowToStructByName code around lines 685-694, and trace how the struct scan definition is generated for each call. The work is complete when definitions are reused per type without changing row-to-struct behavior; use the issue's migration benchmark context to assess whether repeated reflection and allocations improve.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100