koala73 / koala73/worldmonitor

perf(markets): the per-symbol sector tier still proxies through the pinned exit, not the remembered one

Open
#6,279 2 comments 0 reactions 0 assignees View on GitHub
bug P2
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

Split out of the sector exit-rotation work (branch `fix/sector-valuations-rotate-proxy-exit`), where it is marked as a KNOWN GAP in `scripts/_yahoo-sector-valuations.cjs` above the batch fallback.

## What

`collectV7Valuations` runs a per-symbol tier before the batch. That tier tries `direct` first, then falls back to the proxy on the **configured** exit via `resolveProxyString()` — never the exit remembered in Redis. Yahoo's quote fundamentals cache is populated per exit IP, and ~70% of exits omit `trailingPE` for XLK/XLV/XLY/SMH, so on a cycle whose pinned exit is a truncating one those per-symbol proxy requests are spent on symbols they cannot recover.

Two costs, both measured:

- **Requests.** A handful of proxy quote requests per cycle against metered bandwidth. Decodo quota exhaustion has taken the seeder fleet down before.
- **Budget.** More important. One exit costs ~7s through the proxy. Time the per-symbol tier spends is time the rotating batch does not have, and the batch is bounded by `BATCH_BUDGET_MS`. Freeing that budget is what would let rotation reach more exits per cycle.

## Why it was not bundled

The obvious fix — point the per-symbol proxy leg at the remembered exit, or drop it entirely and let everything uncovered fall to the rotating batch — has a real side effect: on healthy cycles the batch would stop running, and the batch is what refreshes the remembered exit and renews its TTL. Any fix needs to keep the preference fresh through some other path.

## Suggested approach

Prefer dropping the per-symbol proxy leg over threading an override into it: it makes the rotating batch the single proxy path, which is simpler than two proxy paths with different exit policies. Pair it with recording the preference from whichever path succeeds.

Found by the review fleet (perf/maintainability) and independently by a gpt-5.5 cross-model pass.

Contributor guide

Open the contributing guide

Research direction

Read scripts/_yahoo-sector-valuations.cjs, starting at collectV7Valuations, resolveProxyString(), and the batch fallback. Trace how the per-symbol proxy leg and rotating batch use the configured or remembered exit, including Redis preference refresh. Done means the redundant proxy cost is addressed without leaving the preference stale or weakening batch rotation within BATCH_BUDGET_MS.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, redis
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.