weaviate / weaviate/weaviate-python-client

Client-side batching: wholesale chunk failures are reported under chunk-local indices, so errors collide and describe the wrong objects

Open
#2,165 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
227
Forks
151
Avg merge
3d 14h
Merged PRs (30d)
11

Description

What happens

With client-side batching, when a whole chunk fails to be sent, BatchObjectReturn.errors
reports fewer failures than actually happened, and the surviving keys describe the wrong
objects.

BatchObjectReturn documents the opposite (weaviate/collections/classes/batch.py, class
docstring):

The keys of the errors and uuids dictionaries will always be equivalent to the
original_index of the objects as you added them to the batching loop

Root cause

The wholesale-failure handler in _BatchBase.__send_batch keys its errors by the position
inside the chunk that was sent:

# weaviate/collections/batch/base.py:648
errors_obj = {
    idx: ErrorObject(message=repr(e), object_=obj) for idx, obj in enumerate(objs)
}

Every other place that fills this dictionary uses the global index — obj.index in
grpc_batch.py:157, sync.py:367/:410, async_.py:409/:452 — because
BatchObjectReturn.__add__ merges chunk results with a plain self.errors.update(other.errors)
(classes/batch.py:231) and does no re-keying. So with a chunk size smaller than the batch,
chunk N re-uses keys 0..len(chunk)-1 and overwrites chunk N-1's entries.

Two observable symptoms:

  1. len(result.errors) under-reports the number of failed objects.
  2. If one chunk succeeds and a later chunk fails wholesale, the successful chunk's uuids
    keys and the failed chunk's errors keys collide and describe different objects
    result.uuids[2] and result.errors[2] refer to two different inputs, so a caller that
    retries "the failed indices" re-inserts rows that already went in and skips rows that did not.

Reproduction

Offline, mock gRPC only, no server and no API keys. Four objects with
fixed_size(batch_size=2), and the mock rejects every object of both chunks (which makes
_BatchGRPC.objects raise WeaviateInsertManyAllFailedError, landing in that except Exception):

result.errors keys : [0, 1]        # 4 objects failed
len(result.errors) : 2
len(failed_objects): 4
errors[0] -> object_.index=2 (input position 2)
errors[1] -> object_.index=3 (input position 3)

And for "first chunk succeeds, second fails wholesale": uuids keys [0, 1] while errors
keys are also [0, 1], describing objects 2 and 3.

Expected: errors keys [0, 1, 2, 3] in the first case and [2, 3] in the second, each
describing the object it names.

Tested on main @ 142d798, Python 3.11.15. I have a patch and mock_tests/ regression tests
for both symptoms, in https://github.com/weaviate/weaviate-python-client/pull/2166

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the wholesale-failure handler in weaviate/collections/batch.py and compare its indexing with the existing paths in grpc_batch.py, sync.py, and async_.py. Run the mock_tests regression cases described in the issue; done means errors retain each object's original global index without collisions or mismatched uuids.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.