l3montree-dev / l3montree-dev/devguard

Replace unbounded IN/Create calls with parameter-safe patterns

Open
#2,911 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
161
Forks
43
Avg merge
1d 8h
Merged PRs (30d)
37

Description

Several places in the codebase use Where("id IN ?", ids) and Create([]Model) on unbounded slices. Postgres' extended protocol caps queries at 65535 parameters total, so these break silently as data grows.

  • Replace Where(...IN...) on slices with Where("id = ANY(?)", pq.Array(ids))
  • Replace Create(largeSlice) with CreateInBatches(largeSlice, batchSize), where batchSize is calculated per-model via shared helper below
const maxPostgresParams = 65535

// calcBatchSize returns a safe batch size for CreateInBatches based on
// the model's column count, with ~10% headroom for extra params
// (e.g. ON CONFLICT clauses in upserts).
func calcBatchSize(db *gorm.DB, model any) (int, error) {
	stmt := &gorm.Statement{DB: db}
	if err := stmt.Parse(model); err != nil {
		return 0, fmt.Errorf("parsing schema: %w", err)
	}

	numFields := len(stmt.Schema.DBNames)
	if numFields == 0 {
		return 0, fmt.Errorf("no db fields found for model %T", model)
	}

	size := int(float64(maxPostgresParams/numFields) * 0.9)
	if size < 1 {
		size = 1
	}
	return size, nil
}

I think we should add semgrep rules to make sure that no new where in or create without batching is used on an array of models. This can replace the implementation of SaveBatchBestEffort which relies on PostgreSQL throwing an error to recursively using a divide and conquer approach.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating all uses of Where("id IN ?", ids), Create on slices, and SaveBatchBestEffort in the Go codebase. Review the proposed calcBatchSize helper and inspect the model schemas involved. Done means replacing unsafe operations with parameter-safe patterns, adding semgrep coverage for regressions, and updating SaveBatchBestEffort without relying on PostgreSQL errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
backend, databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.