iiitl / iiitl/Opensource_Compass

Add TTL Caching to the Setup Guide Handler (Stop Calling the LLM on Every Visit)

Open
#53 0 comments 0 reactions 0 assignees View on GitHub
backend hard performance
Dominant language
TypeScript
Stars
0
Forks
16
PR merge metrics
No merged PRs in 30d

Description

### πŸ“‹ Description

Every time a user visits `/repo/[owner]/[repo]/setup`, the frontend calls the Core Service `GET /repo/setup-guide?repo=owner/repo&user_id=...`. The handler in `setup_guide.go` always:

1. Fetches the repo's README from the GitHub API
2. Makes a full LLM call to the AI service to generate the guide

There is **zero caching**. This means:
- A user who clicks "Setup Guide" and then navigates away and comes back triggers **two full LLM calls**
- Multiple users with the same experience level viewing the same popular repo each trigger individual LLM calls
- LLM API calls cost money and have their own rate limits β€” this design wastes both

The fix is to build a `SetupGuideCache` that caches responses keyed by `(owner, repo, experience_level)` with a TTL. Since README content changes rarely, a TTL of several hours is appropriate.

This is harder than the generic cache cleanup issue (#18 ) because:
- The cache key must combine 3 fields
- The cached value is a complex JSON struct (`SetupGuide`), not a simple string
- You must integrate it cleanly into an existing HTTP handler without breaking the existing code path

---

### πŸ“ Files to Create / Modify

- **Create:** `backend/core_service/internal/cache/setup_guide_cache.go`
- **Modify:** `backend/core_service/internal/handlers/setup_guide.go` β€” check cache before calling AI, write to cache after

---

### πŸ” Current Handler (Always Calls AI)

```go
// setup_guide.go
func (h *SetupGuideHandler) GetSetupGuide(w http.ResponseWriter, r *http.Request) {
// ... parameter validation ...
readme, _ := h.githubClient.FetchReadme(ctx, owner, repoName, token) // GitHub call
guide, _ := h.aiClient.GetSetupGuide(ctx, readme, user.ExperienceLevel) // LLM call ← always happens
json.NewEncoder(w).Encode(map[string]string{"guide": guide})
}
```

---

### βœ… What To Build

**Step 1 β€” Create `backend/core_service/internal/cache/setup_guide_cache.go`:**

```go
package cache

import (
"fmt"
"sync"
"time"
)

type SetupGuideCacheEntry struct {
GuideJSON string // The raw JSON string from the AI
CachedAt time.Time
}

type SetupGuideCache struct {
mu sync.RWMutex
entries map[string]SetupGuideCacheEntry
ttl time.Duration
}

func NewSetupGuideCache(ttl time.Duration) *SetupGuideCache {
c := &SetupGuideCache{
entries: make(map[string]SetupGuideCacheEntry),
ttl: ttl,
}
// Periodically clean up expired entries
go c.startCleanup(ttl * 2)
return c
}

// Key combines owner, repo, and experience level to ensure per-user-type caching
func (c *SetupGuideCache) cacheKey(owner, repo, experienceLevel string) string {
return fmt.Sprintf("%s/%s::%s", owner, repo, experienceLevel)
}

func (c *SetupGuideCache) Get(owner, repo, experienceLevel string) (string, bool) {
key := c.cacheKey(owner, repo, experienceLevel)

c.mu.RLock()
entry, ok := c.entries[key]
c.mu.RUnlock()

if !ok || time.Since(entry.CachedAt) > c.ttl {
return "", false
}
return entry.GuideJSON, true
}

func (c *SetupGuideCache) Set(owner, repo, experienceLevel, guideJSON string) {
key := c.cacheKey(owner, repo, experienceLevel)

c.mu.Lock()
c.entries[key] = SetupGuideCacheEntry{
GuideJSON: guideJSON,
CachedAt: time.Now(),
}
c.mu.Unlock()
}

func (c *SetupGuideCache) startCleanup(interval time.Duration) {
ticker := time.NewTicker(interval)
defer ticker.Stop()
for range ticker.C {
c.mu.Lock()
now := time.Now()
for k, v := range c.entries {
if now.Sub(v.CachedAt) > c.ttl {
delete(c.entries, k)
}
}
c.mu.Unlock()
}
}
```

**Step 2 β€” Add cache to `SetupGuideHandler` struct and constructor:**

```go
// In setup_guide.go:
type SetupGuideHandler struct {
githubClient *clients.GitHubClient
aiClient *clients.AIClient
userRepo *users.Repository
cache *cache.SetupGuideCache // ← add this
}

func NewSetupGuideHandler(gh *clients.GitHubClient, ai *clients.AIClient, ur *users.Repository, c *cache.SetupGuideCache) *SetupGuideHandler {
return &SetupGuideHandler{githubClient: gh, aiClient: ai, userRepo: ur, cache: c}
}
```

**Step 3 β€” Check cache before calling AI, populate cache after:**

```go
func (h *SetupGuideHandler) GetSetupGuide(w http.ResponseWriter, r *http.Request) {
// ... existing parameter validation ...

// ─── Cache Check ────────────────────────────────────
if h.cache != nil {
if cached, ok := h.cache.Get(owner, repoName, user.ExperienceLevel); ok {
log.Printf("SetupGuide: cache HIT for %s/%s (%s)", owner, repoName, user.ExperienceLevel)
w.Header().Set("Content-Type", "application/json")
w.Header().Set("X-Cache", "HIT") // helpful for debugging
w.Write([]byte(`{"guide":` + cached + `}`))
return
}
log.Printf("SetupGuide: cache MISS for %s/%s (%s)", owner, repoName, user.ExperienceLevel)
}

// ─── Existing logic (fetch readme + call AI) ─────────
readme, err := h.githubClient.FetchReadme(r.Context(), owner, repoName, user.GitHubToken)
if err != nil {
http.Error(w, "failed to fetch readme: "+err.Error(), http.StatusInternalServerError)
return
}

guide, err := h.aiClient.GetSetupGuide(r.Context(), readme, user.ExperienceLevel)
if err != nil {
http.Error(w, "failed to generate guide: "+err.Error(), http.StatusInternalServerError)
return
}

// ─── Store in cache ───────────────────────────────────
if h.cache != nil {
h.cache.Set(owner, repoName, user.ExperienceLevel, guide)
}

w.Header().Set("Content-Type", "application/json")
w.Header().Set("X-Cache", "MISS")
json.NewEncoder(w).Encode(map[string]string{"guide": guide})
}
```

**Step 4 β€” Wire it up in `main.go` or `routes.go`:**

```go
guideCache := cache.NewSetupGuideCache(4 * time.Hour) // cached for 4 hours
setupHandler := handlers.NewSetupGuideHandler(githubClient, aiClient, userRepo, guideCache)
```

---

### 🏁 Acceptance Criteria

- [ ] A `SetupGuideCache` with `Get`, `Set`, and background cleanup is implemented
- [ ] Cache key includes `owner`, `repo`, and `experience_level` β€” two users with different levels get different cached responses
- [ ] On a cache HIT, the handler returns immediately without calling the AI service
- [ ] The response includes an `X-Cache: HIT` or `X-Cache: MISS` header (useful for observability)
- [ ] The TTL is configurable β€” passed to `NewSetupGuideCache()` as a parameter, not hardcoded
- [ ] Background cleanup goroutine removes expired entries to prevent unbounded memory growth
- [ ] All code compiles: `cd backend/core_service && go build ./...`
- [ ] Verify manually: hit the setup guide endpoint twice for the same repo, confirm the second response is significantly faster (< 10ms vs potentially seconds)

---

### πŸ’‘ Technical Hints

- The AI service returns a guide as a JSON string (look at `AIClient.GetSetupGuide` return type). Cache the **raw JSON string** β€” don't unmarshal and re-marshal it, that wastes CPU on every cache write/read
- `experienceLevel` is part of the cache key because a "beginner" and an "intermediate" user should get differently-worded guides for the same repo
- The handler currently updates `NewSetupGuideHandler` call β€” find where it's instantiated (likely `main.go` or `routes/`) and update that call to pass the cache

---

### πŸš€ Getting Started

1. Fork the repository
2. Create a branch: `git checkout -b feat/issue-32-setup-guide-cache`
3. Create `backend/core_service/internal/cache/setup_guide_cache.go`
4. Update `setup_guide.go` handler
5. Update constructor call in `main.go`/`routes/`
6. Test: curl the same endpoint twice, 2nd response should be nearly instant
7. Open a PR with `curl` output or logs showing MISS then HIT!

Contributor guide

Open the contributing guide

Research direction

Start with backend/core_service/internal/handlers/setup_guide.go and inspect the SetupGuideHandler constructor and endpoint flow, then create backend/core_service/internal/cache/setup_guide_cache.go. Trace the constructor call in main.go or routes/ and run cd backend/core_service && go build ./.... Done means repeated requests use the configured TTL cache, include the expected X-Cache header, and show MISS then HIT behavior for the same repository and experience level.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.