Kludex / Kludex/awesome-fastapi-projects
Improve dependencies parsing performance
- Dominant language
- TypeScript
- Stars
- 1.6k
- Forks
- 141
- PR merge metrics
- No merged PRs in 30d
Description
The dependencies parsing for repos is quite slow.
The reasons for this are:
- To parse dependencies for repo, a repo needs to be cloned into a temporary directory:
- A temporary directory is created before cloning the repo and destroyed immediately after the parsing is finished
- A new process is spawned for each repo to use a tool to parse the dependencies
- Database operations - updating the repo, creating dependencies, repo-dependencies m2m relations.
There are some heuristics in place to prevent unnecessary parsing:
- A revisions hash is stored from the last repo parsing session - so if the revision does not change, we do nothing.
Possible improvements could be:
- Using a process pool to reuse the processes for dependencies parsing sessions.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the repository dependency-parsing entry point and the code handling temporary clones, parser process creation, and database updates; no files or tests are named. Compare the current parsing flow with process-pool reuse, but the issue provides no benchmark, test, or explicit completion target for deciding when the performance work is done.
Written by the indexing model from the issue text.
Assessment
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100