DIRACGrid / DIRACGrid/diracx

[Jobs Match Making] Scale testing and performance analysis

Open
#874 0 comments 0 reactions 1 assignee Claimed by @devlink42 View on GitHub
Dominant language
Python
Stars
19
Forks
45
Avg merge
4d 23h
Merged PRs (30d)
12

Description

### Description

Push the implementation toward the target scale from the spec: 10M waiting jobs, 200 Hz sustained matching (stretch goal: 1 KHz), ~1000 sites, several hundred tags, non-uniform distributions.

Identify bottlenecks and experiment with Redis configuration tuning. Produce a comparative analysis across all implementations.

Also address robustness:
- Recovery from Redis data loss: rebuild state from a simulated JobDB
- Concurrent matching: verify no duplicate job assignments

### Definition of Done

- [ ] Synthetic dataset at 10M jobs
- [ ] Locust tests with multiple concurrent workers
- [ ] Performance report with plots: throughput and latency at increasing scales
- [ ] Bottleneck analysis
- [ ] Comparison table: all implementations (Python, Redis+Python, Redis+Lua)
- [ ] Rebuild mechanism tested at scale
- [ ] Concurrent matching tested

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.