iiitl / iiitl/Opensource_Compass

Optimize the Watchlist Poller — Deduplicate Repos Watched by Multiple Users

Open Beginner friendly
#15 0 comments 0 reactions 0 assignees View on GitHub
backend good first issue
Dominant language
TypeScript
Stars
0
Forks
16
PR merge metrics
No merged PRs in 30d

Description

### 📋 Description

The notification poller (`poller.go`) checks for new issues by iterating over **every single entry** in the `watched_repos` table. This means if 10 different users all watch the same popular repository (e.g., `facebook/react`), the poller will make **10 separate GitHub API calls** for that same repo — one per user — every poll cycle.

This is wasteful and will hit GitHub API rate limits quickly as the number of users grows. The fix is to deduplicate repos and check each unique repo only once per cycle, then notify all users who watch it.

There is even a `TODO` comment in the code acknowledging this:
```go
// TODO: Fetch distinct repos to avoid duplicate checks if multiple users watch the same repo
```

---

### 📍 File to Change

`backend/core_service/internal/watchlist/poller.go`

---

### 🔍 Current Inefficient Code (lines ~56–103)

```go
// Iterates ALL entries — makes one API call per user per repo
entries, err := p.repo.ListAll(ctx)
for _, entry := range entries {
latestNum, title, issueURL, err := p.githubClient.GetLatestIssue(entry.RepoOwner, entry.RepoName)
// ...notifies user...
}
```

---

### ✅ What To Do

**Step 1:** Group all watchlist entries by `owner/repo` key before the loop:

```go
// Group entries by repo key
type repoGroup struct {
entries []WatchedRepo
}
repoMap := make(map[string][]WatchedRepo)
for _, entry := range entries {
key := entry.RepoOwner + "/" + entry.RepoName
repoMap[key] = append(repoMap[key], entry)
}
```

**Step 2:** Iterate over unique repos — make only **one API call** per repo, then notify all users who watch that repo:

```go
for repoKey, watcherEntries := range repoMap {
// Only one GitHub API call per unique repo
parts := strings.SplitN(repoKey, "/", 2)
latestNum, title, issueURL, err := p.githubClient.GetLatestIssue(parts[0], parts[1])
if err != nil {
log.Printf("Poller: Error fetching %s: %v", repoKey, err)
continue
}

// Notify all users who watch this repo
for _, entry := range watcherEntries {
if latestNum > entry.LatestIssueNumber {
p.repo.UpdateLastChecked(ctx, entry.ID, latestNum)
payload := map[string]interface{}{
"type": "new_issue",
"repo": repoKey,
"issue_number": latestNum,
"issue_title": title,
"issue_url": issueURL,
"message": "New issue detected!",
}
for _, notifier := range p.notifiers {
notifier.NotifyUser(entry.UserID, payload)
}
}
}
}
```

**Step 3:** Remove the `// TODO` comment.

---

### 🏁 Acceptance Criteria

- [ ] The poller makes at most **one GitHub API call** per unique repo per cycle (not one per watcher)
- [ ] All users watching the same repo still receive notifications when a new issue is created
- [ ] The `// TODO` comment is removed
- [ ] Code compiles: `cd backend/core_service && go build ./...`
- [ ] Log messages still print meaningful progress info

---

### 💡 Technical Hints

- `strings.SplitN(repoKey, "/", 2)` splits `"facebook/react"` into `["facebook", "react"]`
- Add `"strings"` to the import block if it's not already there
- The `WatchedRepo` struct is in `model.go` in the same package — check it for field names
- This change reduces GitHub API calls from `O(total_watchers)` to `O(unique_repos)` per cycle

---

### 🚀 Getting Started

1. Fork the repository
2. Create a branch: `git checkout -b fix/issue-16-poller-deduplication`
3. Edit `backend/core_service/internal/watchlist/poller.go`
4. Verify it compiles: `cd backend/core_service && go build ./...`
5. Open a Pull Request!

Contributor guide

Open the contributing guide

Research direction

Start in backend/core_service/internal/watchlist/poller.go, then check model.go for the WatchedRepo fields. Build the service with `cd backend/core_service && go build ./...` after updating the poller. Done means each unique repository is checked once per cycle, all its watchers can still be notified, the TODO is gone, and progress logs remain meaningful.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, go
Domain
api, backend
Issue type
Refactor
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.