Kludex / Kludex/awesome-fastapi-projects

Improve dependencies parsing performance

Open
#36 0 comments 0 reactions 0 assignees View on GitHub
backend chore enhancement
Dominant language
TypeScript
Stars
1.6k
Forks
141
PR merge metrics
No merged PRs in 30d

Description

The dependencies parsing for repos is quite slow.

The reasons for this are:
- To parse dependencies for repo, a repo needs to be cloned into a temporary directory:
- A temporary directory is created before cloning the repo and destroyed immediately after the parsing is finished
- A new process is spawned for each repo to use a tool to parse the dependencies
- Database operations - updating the repo, creating dependencies, repo-dependencies m2m relations.

There are some heuristics in place to prevent unnecessary parsing:
- A revisions hash is stored from the last repo parsing session - so if the revision does not change, we do nothing.

Possible improvements could be:
- Using a process pool to reuse the processes for dependencies parsing sessions.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the repository dependency-parsing entry point and the code handling temporary clones, parser process creation, and database updates; no files or tests are named. Compare the current parsing flow with process-pool reuse, but the issue provides no benchmark, test, or explicit completion target for deciding when the performance work is done.

Written by the indexing model from the issue text.

Assessment

Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.