redis / redis/redis-om-python

[Enhancement] Retrieve multiple records at once with pipeline

Open
#523 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
1.3k
Forks
128
PR merge metrics
No merged PRs in 30d

Description

Currently there is implementation details for saving | deleting multiple records at once using redis pipelines.
Example:

@py_test_mark_asyncio
async def test_delete_many(m):
    member1 = m.Member(
        id=0,
        first_name="Andrew",
        last_name="Brookins",
        email="a@example.com",
        join_date=today,
        age=38,
        bio="This is the user bio.",
    )
    member2 = m.Member(
        id=1,
        first_name="Kim",
        last_name="Brookins",
        email="k@example.com",
        join_date=today,
        age=34,
        bio="This is the bio for Kim.",
    )
    members = [member1, member2]
    result = await m.Member.add(members)
    assert result == [member1, member2]
    result = await m.Member.delete_many(members)
    assert result == 2
    with pytest.raises(NotFoundError):
        await m.Member.get(pk=member1.key())

I want to be able to get multiple records at once, so as to minimize network overhead.
Please make it possible

eg. query_members=[pk1, pk2]

result = await m.Member.get_many(query_members)

So something like the delete_many function:

    @classmethod
    async def get_many(
        cls,
        models: Sequence["RedisModel"],
        pipeline: Optional[redis.client.Pipeline] = None,
    ) -> int:
        db = cls._get_db(pipeline)

        for chunk in ichunked(models, 100):
            pks = [model.key() for model in chunk]
            await cls._get(db, *pks)

        return len(models)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing delete_many implementation and the model retrieval entry points it uses. Define get_many around the requested primary keys and pipeline behavior described in the issue, then add coverage confirming multiple records are retrieved with reduced network overhead and the expected result shape.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, redis
Domain
database
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.