feat: add batch sync for Notion and Website documents in knowledge base
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
## Story / Challenge
When a knowledge base is populated from Notion or Website (web crawl) sources, individual documents can be re-synced from their source via the per-document action menu. However, there is no way to sync multiple documents at once, nor a way to trigger a full sync for all documents in the knowledge base without clicking each one individually. This makes it tedious to keep a large Notion-backed knowledge base up to date after content changes.
## Proposed Solution
**Batch sync (selected documents)**
Add a "Sync" button to the existing batch action bar that appears when one or more documents are selected. The button only appears when at least one selected document is from a Notion or Website source (file-upload documents have no sync concept and are excluded).
**Sync All**
Add a "Sync All" button to the document list header toolbar. It appears for knowledge bases whose `data_source_type` is `notion_import` or `website_crawl`. Clicking it triggers all non-archived Notion/Website documents to sync. The button shows a loading spinner while sync is in progress and shows a success/error toast when complete.
## Technical scope
**Backend** (`api/controllers/console/datasets/data_source.py`)
- Add `GET /datasets//website-sync` endpoint mirroring the existing `GET /datasets//notion/sync`. Both iterate over documents, skip archived ones, and enqueue the appropriate Celery indexing task.
**Frontend** (`web/`)
- `service/knowledge/use-document.ts` – add `useBatchSyncNotion` and `useBatchSyncWebsite` mutation hooks.
- `use-document-selection.ts` – derive `syncableSelectedDocs` (non-archived Notion/Website docs from selection).
- `use-document-actions.ts` – add `handleBatchSync` that calls the appropriate API(s) and triggers `onUpdate`.
- `batch-action.tsx` – add optional `onBatchSync` prop; render Sync button between Re-index and Download.
- `documents-header.tsx` – add optional `onSyncAll` / `isSyncingAll` props; render Sync All button.
- `index.tsx` – implement `handleSyncAll` with polling-based completion detection (polls every 2.5 s up to 5 min, resolves when all documents reach a terminal indexing status).
- i18n – add `batchAction.sync` and `batchAction.syncAll` keys to all 24 locale files.
## Additional Context
The backend sync endpoint for Notion already existed (`DataSourceNotionDatasetSyncApi`). The website equivalent was missing. The frontend had no UI for either endpoint.
- [x] I am willing to submit a PR to address this issue
Contributor guide
Assessment
This issue has not been assessed yet.