weaviate / weaviate/weaviate-python-client
Client-side batching: wholesale chunk failures are reported under chunk-local indices, so errors collide and describe the wrong objects
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 227
- Forks
- 151
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 11
Description
What happens
With client-side batching, when a whole chunk fails to be sent, BatchObjectReturn.errors
reports fewer failures than actually happened, and the surviving keys describe the wrong
objects.
BatchObjectReturn documents the opposite (weaviate/collections/classes/batch.py, class
docstring):
The keys of the
errorsanduuidsdictionaries will always be equivalent to the
original_indexof the objects as you added them to the batching loop
Root cause
The wholesale-failure handler in _BatchBase.__send_batch keys its errors by the position
inside the chunk that was sent:
# weaviate/collections/batch/base.py:648
errors_obj = {
idx: ErrorObject(message=repr(e), object_=obj) for idx, obj in enumerate(objs)
}
Every other place that fills this dictionary uses the global index — obj.index in
grpc_batch.py:157, sync.py:367/:410, async_.py:409/:452 — because
BatchObjectReturn.__add__ merges chunk results with a plain self.errors.update(other.errors)
(classes/batch.py:231) and does no re-keying. So with a chunk size smaller than the batch,
chunk N re-uses keys 0..len(chunk)-1 and overwrites chunk N-1's entries.
Two observable symptoms:
len(result.errors)under-reports the number of failed objects.- If one chunk succeeds and a later chunk fails wholesale, the successful chunk's
uuids
keys and the failed chunk'serrorskeys collide and describe different objects —
result.uuids[2]andresult.errors[2]refer to two different inputs, so a caller that
retries "the failed indices" re-inserts rows that already went in and skips rows that did not.
Reproduction
Offline, mock gRPC only, no server and no API keys. Four objects with
fixed_size(batch_size=2), and the mock rejects every object of both chunks (which makes
_BatchGRPC.objects raise WeaviateInsertManyAllFailedError, landing in that except Exception):
result.errors keys : [0, 1] # 4 objects failed
len(result.errors) : 2
len(failed_objects): 4
errors[0] -> object_.index=2 (input position 2)
errors[1] -> object_.index=3 (input position 3)
And for "first chunk succeeds, second fails wholesale": uuids keys [0, 1] while errors
keys are also [0, 1], describing objects 2 and 3.
Expected: errors keys [0, 1, 2, 3] in the first case and [2, 3] in the second, each
describing the object it names.
Tested on main @ 142d798, Python 3.11.15. I have a patch and mock_tests/ regression tests
for both symptoms, in https://github.com/weaviate/weaviate-python-client/pull/2166
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the wholesale-failure handler in weaviate/collections/batch.py and compare its indexing with the existing paths in grpc_batch.py, sync.py, and async_.py. Run the mock_tests regression cases described in the issue; done means errors retain each object's original global index without collisions or mismatched uuids.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 15/100