googleapis / googleapis/python-aiplatform

gcp vertex ai engine concurrent import requests limit makes rag.import_files_async call almost pointless if uploading to multiple corpora

Đang mở
#5,424 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
api: vertex-ai
Ngôn ngữ chính
Python
Star
905
Fork
465
Merge trung bình
1 ngày 13 giờ
Pull request đã merge (30 ngày)
44

Mô tả

The code below works to get the first 50 files from `bucket_dir_1` and then does not get anymore, no does it get any other files at all from any of the other `bucket_dirs`. the vertex ai rag engine says that up to 10K files can be imported at one time so I've no idea why I'd be hitting a limit.

```
corpora_ids_and_display_names = [('bucket_dir_1', 'projects/$PROJECT_NUMBER/locations/us-central1/ragCorpora/$CORPUS_ID_1'), ('bucket_dir_2', 'projects/$PROJECT_NUMBER/locations/us-central1/ragCorpora/$CORPUS_ID_2')]

async def import_files_async(corpus_id_and_display_name: tuple):
try:
response = await rag.import_files_async(
corpus_name=corpus_id_and_display_name[1],
paths=[f"gs://some-bucket/{corpus_id_and_display_name[0]}"],
)
return response
except Exception as e:
print(f"error: {e}")

async def create_import_files_task_group_async():
coros = [import_files_async(corpus_id_and_display_name) for corpus_id_and_display_name in corpora_ids_and_display_names]
import_files_task_group = [asyncio.create_task(coro) for coro in coros]
results = await asyncio.gather(*import_files_task_group, return_exceptions=True)
print(f"results: {results}")

asyncio.run(create_import_files_task_group_async())
```

```
# final asyncio.gather/asyncio.run response -- no errors
results: [, , , , ]
```

```
# works to list corpora hence corpora paths are correct
for corpus_id_and_display_name in corpora_ids_and_display_names:
print(rag.list_files(corpus_name=corpus_id_and_display_name[1]))
```

```
# works to list all files in each bucket path hence import_files_async
# has valid paths argument
for corpus_id_and_display_name in corpora_ids_and_display_names:
subprocess.run(['gcloud storage ls', f"gs://bucket/{corpus_id_and_display_name[0]}"])
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.