l3montree-dev / l3montree-dev/devguard
Replace unbounded IN/Create calls with parameter-safe patterns
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 161
- Forks
- 43
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 37
Description
Several places in the codebase use Where("id IN ?", ids) and Create([]Model) on unbounded slices. Postgres' extended protocol caps queries at 65535 parameters total, so these break silently as data grows.
- Replace
Where(...IN...)on slices withWhere("id = ANY(?)", pq.Array(ids)) - Replace
Create(largeSlice)withCreateInBatches(largeSlice, batchSize), wherebatchSizeis calculated per-model via shared helper below
const maxPostgresParams = 65535
// calcBatchSize returns a safe batch size for CreateInBatches based on
// the model's column count, with ~10% headroom for extra params
// (e.g. ON CONFLICT clauses in upserts).
func calcBatchSize(db *gorm.DB, model any) (int, error) {
stmt := &gorm.Statement{DB: db}
if err := stmt.Parse(model); err != nil {
return 0, fmt.Errorf("parsing schema: %w", err)
}
numFields := len(stmt.Schema.DBNames)
if numFields == 0 {
return 0, fmt.Errorf("no db fields found for model %T", model)
}
size := int(float64(maxPostgresParams/numFields) * 0.9)
if size < 1 {
size = 1
}
return size, nil
}
I think we should add semgrep rules to make sure that no new where in or create without batching is used on an array of models. This can replace the implementation of SaveBatchBestEffort which relies on PostgreSQL throwing an error to recursively using a divide and conquer approach.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating all uses of Where("id IN ?", ids), Create on slices, and SaveBatchBestEffort in the Go codebase. Review the proposed calcBatchSize helper and inspect the model schemas involved. Done means replacing unsafe operations with parameter-safe patterns, adding semgrep coverage for regressions, and updating SaveBatchBestEffort without relying on PostgreSQL errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql
- Domain
- backend, databases
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100