[Jobs Match Making] Scale testing and performance analysis
- Dominant language
- Python
- Stars
- 19
- Forks
- 45
- Avg merge
- 4d 23h
- Merged PRs (30d)
- 12
Description
### Description
Push the implementation toward the target scale from the spec: 10M waiting jobs, 200 Hz sustained matching (stretch goal: 1 KHz), ~1000 sites, several hundred tags, non-uniform distributions.
Identify bottlenecks and experiment with Redis configuration tuning. Produce a comparative analysis across all implementations.
Also address robustness:
- Recovery from Redis data loss: rebuild state from a simulated JobDB
- Concurrent matching: verify no duplicate job assignments
### Definition of Done
- [ ] Synthetic dataset at 10M jobs
- [ ] Locust tests with multiple concurrent workers
- [ ] Performance report with plots: throughput and latency at increasing scales
- [ ] Bottleneck analysis
- [ ] Comparison table: all implementations (Python, Redis+Python, Redis+Lua)
- [ ] Rebuild mechanism tested at scale
- [ ] Concurrent matching tested
Contributor guide
Assessment
This issue has not been assessed yet.