googleapis / googleapis/python-aiplatform

gcp vertex ai engine concurrent import requests limit makes rag.import_files_async call almost pointless if uploading to multiple corpora

Open
#5,424 1 comment 0 reactions 0 assignees View on GitHub
api: vertex-ai
Dominant language
Python
Stars
905
Forks
465
Avg merge
1d 13h
Merged PRs (30d)
44

Description

The code below works to get the first 50 files from `bucket_dir_1` and then does not get anymore, no does it get any other files at all from any of the other `bucket_dirs`. the vertex ai rag engine says that up to 10K files can be imported at one time so I've no idea why I'd be hitting a limit.

```
corpora_ids_and_display_names = [('bucket_dir_1', 'projects/$PROJECT_NUMBER/locations/us-central1/ragCorpora/$CORPUS_ID_1'), ('bucket_dir_2', 'projects/$PROJECT_NUMBER/locations/us-central1/ragCorpora/$CORPUS_ID_2')]

async def import_files_async(corpus_id_and_display_name: tuple):
try:
response = await rag.import_files_async(
corpus_name=corpus_id_and_display_name[1],
paths=[f"gs://some-bucket/{corpus_id_and_display_name[0]}"],
)
return response
except Exception as e:
print(f"error: {e}")

async def create_import_files_task_group_async():
coros = [import_files_async(corpus_id_and_display_name) for corpus_id_and_display_name in corpora_ids_and_display_names]
import_files_task_group = [asyncio.create_task(coro) for coro in coros]
results = await asyncio.gather(*import_files_task_group, return_exceptions=True)
print(f"results: {results}")

asyncio.run(create_import_files_task_group_async())
```

```
# final asyncio.gather/asyncio.run response -- no errors
results: [, , , , ]
```

```
# works to list corpora hence corpora paths are correct
for corpus_id_and_display_name in corpora_ids_and_display_names:
print(rag.list_files(corpus_name=corpus_id_and_display_name[1]))
```

```
# works to list all files in each bucket path hence import_files_async
# has valid paths argument
for corpus_id_and_display_name in corpora_ids_and_display_names:
subprocess.run(['gcloud storage ls', f"gs://bucket/{corpus_id_and_display_name[0]}"])
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the provided asyncio.gather and asyncio.run example with rag.import_files_async across multiple corpus paths. Compare the returned AsyncOperation objects with the files listed in each corpus, and check whether the operations expose errors or completion results. Done means identifying why concurrent imports stop after the first 50 files and documenting or fixing the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.