Improve performance with lots of jobs
Open
- Dominant language
- Python
- Stars
- 6
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Several operations in Polyphemus are linear in the total job count. For example, to find the latest job, you have to list all the jobs on the filesystem, open their JSON documents, and sort the result.
It would be great to someday not slow down too much in the presence of lots of old jobs, perhaps by caching some metadata in an index.
(Expanded from #1.)
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files or tests; begin by locating the job-listing, JSON-loading, and latest-job sorting paths. Define and measure an index or cache that avoids scanning all old jobs, then verify that job behavior remains correct with many jobs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100