lablup / lablup/backend.ai

Determine the maximum item count limit for bulk operations

Open
#11,938 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Bulk operations (bulk_\*) process items individually with partial-failure support. Without an upper bound on the number of items per request, a single bulk call can cause excessive DB load, long-running transactions, and degraded responsiveness.

This task is to determine an appropriate maximum item count limit for bulk operations through testing and validation.

Scope:

- Benchmark bulk operations (create/upsert/update/purge) at increasing item counts to find performance and stability thresholds.
- Identify the practical maximum item count per bulk request based on measured DB load, latency, and transaction behavior.
- Recommend a concrete limit value to enforce at the API/service layer.

JIRA Issue: BA-6281

Contributor guide

Open the contributing guide

Research direction

Start by locating the bulk_* API/service handlers and their create, upsert, update, and purge paths. Benchmark increasing request sizes while recording DB load, latency, and transaction behavior. Done means identifying a measured stability threshold and documenting a concrete maximum item count for enforcement at the API/service layer.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.