adafruit / adafruit/cookiecutter-adafruit-circuitpython

pre-commit pylint checks are doing too much work

Open
#121 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
27
Forks
39
PR merge metrics
No merged PRs in 30d

Description

I noticed that in adafruit_datetime, the pre-commit check could take a long time, especially for the "tests" step. Furthermore, all 4 of my CPU cores were active.

I believe this is because by default, pre-commit
* assumes that the program accepts filenames on its commandline/argv
* runs multiple processes in parallel if there are multiple files

Since pylint needs to get a view of all the source files it's checking in order to do proper code duplication checks, we make our own list of files to pylint with `find` and ignore the positional arguments that are given. But unless we also specify `pass_filenames: false` pre-commit doesn't know about it and starts invoking the "pylint all files" command once for each file!

This change is one I'm testing locally in adafruit_datetime:
```
From 10ebce0339182073dc0aabd010f73df00ae348a6 Mon Sep 17 00:00:00 2001
From: Jeff Epler
Date: Mon, 22 Mar 2021 20:34:25 -0500
Subject: [PATCH] pre-commit: don't do too much work

---
.pre-commit-config.yaml | 2 ++
1 file changed, 2 insertions(+)

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index cce4c7b..96f0626 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -32,6 +32,7 @@ repos:
entry: /usr/bin/env bash -c
args: ['([[ ! -d "examples" ]] || for example in $(find . -path "./examples/*.py"); do pylint --disable=missing-docstring,invalid-name $example; done)']
language: system
+ pass_filenames: false
- repo: local
hooks:
- id: pylint_tests
@@ -40,3 +41,4 @@ repos:
entry: /usr/bin/env bash -c
args: ['([[ ! -d "tests" ]] || for test in $(find . -path "./tests/*.py"); do pylint --disable=missing-docstring $test; done)']
language: system
+ pass_filenames: false
--
2.29.2

```

If we want to make a change like this we'll have to apply it with adabot to existing repos as well.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in .pre-commit-config.yaml, especially the local pylint hooks for examples and tests and their find-based entries. Run pre-commit on a generated library to confirm each pylint hook runs once rather than once per file; done when the template change can be propagated to existing repositories with adabot.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, tooling
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.