backoff strategy should be used for rate-limited errors on milvus or reducing batch_size config

It's recurrent to see the following type of errors on non-local setups:

pymilvus.exceptions.MilvusException: <MilvusException: (code=49, message=Retry run out of 10 retry times, message=request is rejected by grpc RateLimiter middleware, please retry later, req: /milvus.proto.milvus.MilvusService/Insert)>

Full traceback:

Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/ubuntu/vector-db-benchmark/engine/base_client/upload.py", line 90, in _upload_batch
    cls.upload_batch(ids, vectors, metadata)
  File "/home/ubuntu/vector-db-benchmark/engine/clients/milvus/upload.py", line 68, in upload_batch
    cls.upload_with_backoff(field_values, ids, vectors)
  File "/usr/local/lib/python3.10/dist-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/ubuntu/vector-db-benchmark/engine/clients/milvus/upload.py", line 75, in upload_with_backoff
    cls.collection.insert([ids, vectors] + field_values)
  File "/usr/local/lib/python3.10/dist-packages/pymilvus/orm/collection.py", line 443, in insert
    res = conn.batch_insert(self._name, entities, partition_name,
  File "/usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py", line 109, in handler
    raise e
  File "/usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py", line 105, in handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py", line 136, in handler
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py", line 80, in handler
    raise MilvusException(e.code, f"{timeout_msg}, message={e.message}") from e
pymilvus.exceptions.MilvusException: <MilvusException: (code=49, message=Retry run out of 10 retry times, message=request is rejected by grpc RateLimiter middleware, please retry later, req: /milvus.proto.milvus.MilvusService/Insert)>
"""


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "/home/ubuntu/vector-db-benchmark/run.py", line 79, in <module>
    app()

  File "/home/ubuntu/vector-db-benchmark/run.py", line 74, in run
    raise e

  File "/home/ubuntu/vector-db-benchmark/run.py", line 52, in run
    client.run_experiment(dataset, skip_upload, skip_search)

  File "/home/ubuntu/vector-db-benchmark/engine/base_client/client.py", line 70, in run_experiment
    upload_stats = self.uploader.upload(

  File "/home/ubuntu/vector-db-benchmark/engine/base_client/upload.py", line 56, in upload
    latencies = list(

  File "/usr/lib/python3.10/multiprocessing/pool.py", line 873, in next
    raise value

pymilvus.exceptions.MilvusException: <MilvusException: (code=49, message=Retry run out of 10 retry times, message=request is rejected by grpc RateLimiter middleware, please retry later, req: /milvus.proto.milvus.MilvusService/Insert)>

Given milvus configs dont specify the batch_size we're using 64 vectors, which seems to be constantly making the error state above.
I suggest to either respect API Rate Limits With a Backoff or reduce the batch size.

Start with engine/clients/milvus/upload.py, especially upload_with_backoff and the collection.insert call, then trace how batches are created in engine/base_client/upload.py. Check the Milvus configuration and run.py entry point to reproduce the rate-limit failure. Done means uploads handle the reported rate-limited error with backoff or use a reduced batch size without recurring failures.