mpfaffenberger / mpfaffenberger/code_puppy

file_operations._list_files: O(n^2) duplicate-dir scan, 4-5 syscalls per entry, double sort; ignore pattern '**/.*' contradicts its own comment

Open
#440 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
814
Forks
278
Avg merge
2d 5h
Merged PRs (30d)
76

Description

File: code_puppy/tools/file_operations.py - _list_files (~lines 150-300)

Severity: Medium (performance + correctness)

  1. O(n^2) duplicate-directory check. For every file returned by ripgrep, the code synthesizes parent directory entries and checks for duplicates with a linear scan over the entire results list, inside a loop over path components:
for i in range(len(path_parts)):
    partial_path = os.sep.join(path_parts[: i + 1])
    if not any(f.path == partial_path and f.type == "directory" for f in results):
        results.append(...)

On a repo with tens of thousands of files this is quadratic (each any() walks all accumulated entries). Fix: keep a seen_dirs: set[str] alongside results - O(1) membership.

  1. Redundant stat syscalls per entry: os.path.exists, os.path.isfile/isdir, os.path.getsize, then os.stat again - 4-5 syscalls per file where one os.stat (with try/except) provides everything.

  2. Results are sorted twice with the same _sort_key (once to build file_entries for the UI message, once to build the text output, ~lines 415 and 440). Sort once into a local.

  3. should_ignore_path ignores ALL hidden files by design confusion (code_puppy/tools/common.py:437): the pattern list contains "**/.*" with the comment "# Commented out as it might be too aggressive" - but it is NOT commented out. Either remove the pattern or fix the comment; right now the comment actively misleads.

  4. _list_files and _grep duplicate the rg-discovery block (shutil.which + venv probing) and the tempfile-ignore-file dance verbatim - extract a _find_ripgrep() and _write_ignore_file() helper (also reduces the two separate temp-file cleanup paths).

Filed by Zen Reviewer B (code-puppy-60635a)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in code_puppy/tools/file_operations.py at _list_files and _grep, then inspect should_ignore_path in code_puppy/tools/common.py. Trace the ripgrep discovery, temporary ignore-file handling, directory-entry construction, metadata lookup, and output sorting. Done means the listed performance issues are addressed, shared discovery and cleanup logic is extracted, and the '**/.*' comment and behavior agree.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.