[Enhancement] Retrieve multiple records at once with pipeline
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 128
- PR merge metrics
- No merged PRs in 30d
Description
Currently there is implementation details for saving | deleting multiple records at once using redis pipelines.
Example:
@py_test_mark_asyncio
async def test_delete_many(m):
member1 = m.Member(
id=0,
first_name="Andrew",
last_name="Brookins",
email="a@example.com",
join_date=today,
age=38,
bio="This is the user bio.",
)
member2 = m.Member(
id=1,
first_name="Kim",
last_name="Brookins",
email="k@example.com",
join_date=today,
age=34,
bio="This is the bio for Kim.",
)
members = [member1, member2]
result = await m.Member.add(members)
assert result == [member1, member2]
result = await m.Member.delete_many(members)
assert result == 2
with pytest.raises(NotFoundError):
await m.Member.get(pk=member1.key())
I want to be able to get multiple records at once, so as to minimize network overhead.
Please make it possible
eg. query_members=[pk1, pk2]
result = await m.Member.get_many(query_members)
So something like the delete_many function:
@classmethod
async def get_many(
cls,
models: Sequence["RedisModel"],
pipeline: Optional[redis.client.Pipeline] = None,
) -> int:
db = cls._get_db(pipeline)
for chunk in ichunked(models, 100):
pks = [model.key() for model in chunk]
await cls._get(db, *pks)
return len(models)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing delete_many implementation and the model retrieval entry points it uses. Define get_many around the requested primary keys and pipeline behavior described in the issue, then add coverage confirming multiple records are retrieved with reduced network overhead and the expected result shape.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, redis
- Domain
- database
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100