NatLabRockies / NatLabRockies/plexosdb
Standardize chunking policy for bulk APIs and IN-clause lookups
Open
@mcllerena is already working on this.
Since May 7, 2026.
- Dominant language
- Python
- Stars
- 37
- Forks
- 21
- Avg merge
- 1h 23m
- Merged PRs (30d)
- 3
Description
Problem
Chunking is currently handled in two different ways:
- User-facing
chunksizeparams for write batching (for example in bulk insert APIs). - Method-local internal constants like
CHUNK = 900for safeIN (...)query parameter binding.
Both are valid, but behavior and naming are not standardized, which makes reviews and maintenance harder.
Why this matters
- Inconsistent patterns across methods increase review friction.
- Safe bind limits can be missed in new bulk paths.
- It is not obvious when a knob is user-facing vs internal safety.
Proposal
Define and document a single policy:
- Keep
chunksizeas a per-method user-facing write batching parameter. - Keep bind-safety chunking internal (not user-facing).
- Introduce a shared internal constant/helper for
IN (...)chunking (instead of ad-hoc local constants). - Update affected methods to follow the same pattern.
Candidate code paths
src/plexosdb/db.py(get_memberships_systemand otherIN (...)lookups)src/plexosdb/utils.py(insert_property_valuesmembership-name lookup)- New/ongoing bulk APIs (for example
add_attributes_from_records)
Acceptance criteria
- Policy documented in developer docs/contributing notes
- Shared internal helper or constant introduced
- Existing bulk/query methods aligned
- Tests cover large inputs across chunk boundaries
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.