googleapis / googleapis/python-aiplatform
gcp vertex ai engine concurrent import requests limit makes rag.import_files_async call almost pointless if uploading to multiple corpora
- Dominant language
- Python
- Stars
- 905
- Forks
- 465
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 44
Description
The code below works to get the first 50 files from `bucket_dir_1` and then does not get anymore, no does it get any other files at all from any of the other `bucket_dirs`. the vertex ai rag engine says that up to 10K files can be imported at one time so I've no idea why I'd be hitting a limit.
```
corpora_ids_and_display_names = [('bucket_dir_1', 'projects/$PROJECT_NUMBER/locations/us-central1/ragCorpora/$CORPUS_ID_1'), ('bucket_dir_2', 'projects/$PROJECT_NUMBER/locations/us-central1/ragCorpora/$CORPUS_ID_2')]
async def import_files_async(corpus_id_and_display_name: tuple):
try:
response = await rag.import_files_async(
corpus_name=corpus_id_and_display_name[1],
paths=[f"gs://some-bucket/{corpus_id_and_display_name[0]}"],
)
return response
except Exception as e:
print(f"error: {e}")
async def create_import_files_task_group_async():
coros = [import_files_async(corpus_id_and_display_name) for corpus_id_and_display_name in corpora_ids_and_display_names]
import_files_task_group = [asyncio.create_task(coro) for coro in coros]
results = await asyncio.gather(*import_files_task_group, return_exceptions=True)
print(f"results: {results}")
asyncio.run(create_import_files_task_group_async())
```
```
# final asyncio.gather/asyncio.run response -- no errors
results: [, , , , ]
```
```
# works to list corpora hence corpora paths are correct
for corpus_id_and_display_name in corpora_ids_and_display_names:
print(rag.list_files(corpus_name=corpus_id_and_display_name[1]))
```
```
# works to list all files in each bucket path hence import_files_async
# has valid paths argument
for corpus_id_and_display_name in corpora_ids_and_display_names:
subprocess.run(['gcloud storage ls', f"gs://bucket/{corpus_id_and_display_name[0]}"])
```
Contributor guide
Research direction
Start by reproducing the provided asyncio.gather and asyncio.run example with rag.import_files_async across multiple corpus paths. Compare the returned AsyncOperation objects with the files listed in each corpus, and check whether the operations expose errors or completion results. Done means identifying why concurrent imports stop after the first 50 files and documenting or fixing the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100