[APIView] staging sync pipeline times out due to large container sequential processing
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 115
Description
The `apiview-sync-to-dev` pipeline consistently times out when syncing Cosmos DB data from production backups to staging environments. The pipeline has a default 60-minute timeout, but the sync process takes much longer.
## Root Cause
The sync script ([sync_cosmosdb.py](https://github.com/Azure/azure-sdk-tools/blob/main/eng/scripts/python/apiview-syncdb/sync_cosmosdb.py)) has two performance issues:
### 1. Sequential upserts (one record at a time)
```python
for row in missing_records:
dest_container_client.upsert_item(row) # ~80ms per record
```
### 2. Large containers processed first
```python
COSMOS_CONTAINERS = ["APIRevisions", "Reviews", "Comments", "PullRequests", "SamplesRevisions", "Permissions", "Projects"]
```
The pipeline times out before reaching smaller containers like `Permissions` and `Projects`.
## Impact
- New containers added to the sync list may never get synced
- Staging environments fall out of sync with production
- Manual intervention required to sync data
## Suggested Improvements
### Short-term
- [ ] Increase pipeline timeout ([apiview-sync-to-dev.yml](https://github.com/Azure/azure-sdk-tools/blob/main/src/dotnet/APIView/apiview-sync-to-dev.yml)) to 3+ hours
- [ ] Reorder containers to process smaller ones first
- [ ] Add progress logging to track sync status
### Long-term
- [ ] Implement batch/parallel upserts instead of sequential
- [ ] Add incremental sync optimization (skip containers with no changes)
- [ ] Consider using Cosmos DB bulk executor for large datasets
- [ ] Split large containers into separate pipeline jobs
## Related
- Pipeline: `apiview-sync-to-dev`
- Script: `eng/scripts/python/apiview-syncdb/sync_cosmosdb.py`
- YAML: `src/dotnet/APIView/apiview-sync-to-dev.yml`
Contributor guide
Assessment
This issue has not been assessed yet.