iiitl / iiitl/Opensource_Compass
Optimize the Watchlist Poller — Deduplicate Repos Watched by Multiple Users
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
### 📋 Description
The notification poller (`poller.go`) checks for new issues by iterating over **every single entry** in the `watched_repos` table. This means if 10 different users all watch the same popular repository (e.g., `facebook/react`), the poller will make **10 separate GitHub API calls** for that same repo — one per user — every poll cycle.
This is wasteful and will hit GitHub API rate limits quickly as the number of users grows. The fix is to deduplicate repos and check each unique repo only once per cycle, then notify all users who watch it.
There is even a `TODO` comment in the code acknowledging this:
```go
// TODO: Fetch distinct repos to avoid duplicate checks if multiple users watch the same repo
```
---
### 📍 File to Change
`backend/core_service/internal/watchlist/poller.go`
---
### 🔍 Current Inefficient Code (lines ~56–103)
```go
// Iterates ALL entries — makes one API call per user per repo
entries, err := p.repo.ListAll(ctx)
for _, entry := range entries {
latestNum, title, issueURL, err := p.githubClient.GetLatestIssue(entry.RepoOwner, entry.RepoName)
// ...notifies user...
}
```
---
### ✅ What To Do
**Step 1:** Group all watchlist entries by `owner/repo` key before the loop:
```go
// Group entries by repo key
type repoGroup struct {
entries []WatchedRepo
}
repoMap := make(map[string][]WatchedRepo)
for _, entry := range entries {
key := entry.RepoOwner + "/" + entry.RepoName
repoMap[key] = append(repoMap[key], entry)
}
```
**Step 2:** Iterate over unique repos — make only **one API call** per repo, then notify all users who watch that repo:
```go
for repoKey, watcherEntries := range repoMap {
// Only one GitHub API call per unique repo
parts := strings.SplitN(repoKey, "/", 2)
latestNum, title, issueURL, err := p.githubClient.GetLatestIssue(parts[0], parts[1])
if err != nil {
log.Printf("Poller: Error fetching %s: %v", repoKey, err)
continue
}
// Notify all users who watch this repo
for _, entry := range watcherEntries {
if latestNum > entry.LatestIssueNumber {
p.repo.UpdateLastChecked(ctx, entry.ID, latestNum)
payload := map[string]interface{}{
"type": "new_issue",
"repo": repoKey,
"issue_number": latestNum,
"issue_title": title,
"issue_url": issueURL,
"message": "New issue detected!",
}
for _, notifier := range p.notifiers {
notifier.NotifyUser(entry.UserID, payload)
}
}
}
}
```
**Step 3:** Remove the `// TODO` comment.
---
### 🏁 Acceptance Criteria
- [ ] The poller makes at most **one GitHub API call** per unique repo per cycle (not one per watcher)
- [ ] All users watching the same repo still receive notifications when a new issue is created
- [ ] The `// TODO` comment is removed
- [ ] Code compiles: `cd backend/core_service && go build ./...`
- [ ] Log messages still print meaningful progress info
---
### 💡 Technical Hints
- `strings.SplitN(repoKey, "/", 2)` splits `"facebook/react"` into `["facebook", "react"]`
- Add `"strings"` to the import block if it's not already there
- The `WatchedRepo` struct is in `model.go` in the same package — check it for field names
- This change reduces GitHub API calls from `O(total_watchers)` to `O(unique_repos)` per cycle
---
### 🚀 Getting Started
1. Fork the repository
2. Create a branch: `git checkout -b fix/issue-16-poller-deduplication`
3. Edit `backend/core_service/internal/watchlist/poller.go`
4. Verify it compiles: `cd backend/core_service && go build ./...`
5. Open a Pull Request!
Contributor guide
Research direction
Start in backend/core_service/internal/watchlist/poller.go, then check model.go for the WatchedRepo fields. Build the service with `cd backend/core_service && go build ./...` after updating the poller. Done means each unique repository is checked once per cycle, all its watchers can still be notified, the TODO is gone, and progress logs remain meaningful.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, go
- Domain
- api, backend
- Issue type
- Refactor
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100