koala73 / koala73/worldmonitor
perf(markets): the per-symbol sector tier still proxies through the pinned exit, not the remembered one
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
Split out of the sector exit-rotation work (branch `fix/sector-valuations-rotate-proxy-exit`), where it is marked as a KNOWN GAP in `scripts/_yahoo-sector-valuations.cjs` above the batch fallback.
## What
`collectV7Valuations` runs a per-symbol tier before the batch. That tier tries `direct` first, then falls back to the proxy on the **configured** exit via `resolveProxyString()` — never the exit remembered in Redis. Yahoo's quote fundamentals cache is populated per exit IP, and ~70% of exits omit `trailingPE` for XLK/XLV/XLY/SMH, so on a cycle whose pinned exit is a truncating one those per-symbol proxy requests are spent on symbols they cannot recover.
Two costs, both measured:
- **Requests.** A handful of proxy quote requests per cycle against metered bandwidth. Decodo quota exhaustion has taken the seeder fleet down before.
- **Budget.** More important. One exit costs ~7s through the proxy. Time the per-symbol tier spends is time the rotating batch does not have, and the batch is bounded by `BATCH_BUDGET_MS`. Freeing that budget is what would let rotation reach more exits per cycle.
## Why it was not bundled
The obvious fix — point the per-symbol proxy leg at the remembered exit, or drop it entirely and let everything uncovered fall to the rotating batch — has a real side effect: on healthy cycles the batch would stop running, and the batch is what refreshes the remembered exit and renews its TTL. Any fix needs to keep the preference fresh through some other path.
## Suggested approach
Prefer dropping the per-symbol proxy leg over threading an override into it: it makes the rotating batch the single proxy path, which is simpler than two proxy paths with different exit policies. Pair it with recording the preference from whichever path succeeds.
Found by the review fleet (perf/maintainability) and independently by a gpt-5.5 cross-model pass.
Contributor guide
Research direction
Read scripts/_yahoo-sector-valuations.cjs, starting at collectV7Valuations, resolveProxyString(), and the batch fallback. Trace how the per-symbol proxy leg and rotating batch use the configured or remembered exit, including Redis preference refresh. Done means the redundant proxy cost is addressed without leaving the preference stale or weakening batch rotation within BATCH_BUDGET_MS.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, redis
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100