NatLabRockies / NatLabRockies/plexosdb

Standardize chunking policy for bulk APIs and IN-clause lookups

Open
#132 0 comments 0 reactions 1 assignee View on GitHub

@mcllerena is already working on this.

Since May 7, 2026.

Dominant language
Python
Stars
37
Forks
21
Avg merge
1h 23m
Merged PRs (30d)
3

Description

Problem

Chunking is currently handled in two different ways:

  1. User-facing chunksize params for write batching (for example in bulk insert APIs).
  2. Method-local internal constants like CHUNK = 900 for safe IN (...) query parameter binding.

Both are valid, but behavior and naming are not standardized, which makes reviews and maintenance harder.

Why this matters

  • Inconsistent patterns across methods increase review friction.
  • Safe bind limits can be missed in new bulk paths.
  • It is not obvious when a knob is user-facing vs internal safety.

Proposal

Define and document a single policy:

  • Keep chunksize as a per-method user-facing write batching parameter.
  • Keep bind-safety chunking internal (not user-facing).
  • Introduce a shared internal constant/helper for IN (...) chunking (instead of ad-hoc local constants).
  • Update affected methods to follow the same pattern.

Candidate code paths

  • src/plexosdb/db.py (get_memberships_system and other IN (...) lookups)
  • src/plexosdb/utils.py (insert_property_values membership-name lookup)
  • New/ongoing bulk APIs (for example add_attributes_from_records)

Acceptance criteria

  • Policy documented in developer docs/contributing notes
  • Shared internal helper or constant introduced
  • Existing bulk/query methods aligned
  • Tests cover large inputs across chunk boundaries

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.