Azure / Azure/azure-sdk-tools

[APIView] staging sync pipeline times out due to large container sequential processing

Open
#14,014 0 comments 1 reaction 2 assignees Claimed by @AlitzelMendez View on GitHub
needs-triage
Dominant language
C#
Stars
135
Forks
260
Avg merge
1d 22h
Merged PRs (30d)
115

Description

The `apiview-sync-to-dev` pipeline consistently times out when syncing Cosmos DB data from production backups to staging environments. The pipeline has a default 60-minute timeout, but the sync process takes much longer.

## Root Cause
The sync script ([sync_cosmosdb.py](https://github.com/Azure/azure-sdk-tools/blob/main/eng/scripts/python/apiview-syncdb/sync_cosmosdb.py)) has two performance issues:

### 1. Sequential upserts (one record at a time)
```python
for row in missing_records:
dest_container_client.upsert_item(row) # ~80ms per record
```

### 2. Large containers processed first
```python
COSMOS_CONTAINERS = ["APIRevisions", "Reviews", "Comments", "PullRequests", "SamplesRevisions", "Permissions", "Projects"]
```

The pipeline times out before reaching smaller containers like `Permissions` and `Projects`.

## Impact
- New containers added to the sync list may never get synced
- Staging environments fall out of sync with production
- Manual intervention required to sync data

## Suggested Improvements

### Short-term
- [ ] Increase pipeline timeout ([apiview-sync-to-dev.yml](https://github.com/Azure/azure-sdk-tools/blob/main/src/dotnet/APIView/apiview-sync-to-dev.yml)) to 3+ hours
- [ ] Reorder containers to process smaller ones first
- [ ] Add progress logging to track sync status

### Long-term
- [ ] Implement batch/parallel upserts instead of sequential
- [ ] Add incremental sync optimization (skip containers with no changes)
- [ ] Consider using Cosmos DB bulk executor for large datasets
- [ ] Split large containers into separate pipeline jobs

## Related
- Pipeline: `apiview-sync-to-dev`
- Script: `eng/scripts/python/apiview-syncdb/sync_cosmosdb.py`
- YAML: `src/dotnet/APIView/apiview-sync-to-dev.yml`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.