dlt-hub / dlt-hub/dlt

repo: use `pre-commit`

Open
#3,224 5 comments 0 reactions 0 assignees View on GitHub
tech-debt
Dominant language
Python
Stars
5.9k
Forks
605
Avg merge
1d 14h
Merged PRs (30d)
38

Description

[pre-commit](https://github.com/pre-commit/pre-commit) is a Python dev tool to manage git hooks (a git built-in feature). Primarily, this would allow to "force run" linting and formatting steps before pushing code.

## Current problems and solutions
### Forgetting to lint & format
Often, we push to a PR -> trigger CI -> linting step fails -> go back to your IDE to fix it.

consequences:
1. this is slower than running the check locally -> fixing it -> pushing
2. this eats CI resource away, slowing CI for everyone
3. this clutters the commit history with "fix formatting" commits. Also, these format fixes might not relate to the immediate previous commit, which entangles changes

### Irrelevant files included in linting
Current usage of `make lint` catches all files in the repository. This catches debugging code changes or edits to examples / notebooks / etc. that are not relevant to the current commit / PR. I often have to `git stash push -m "stash to lint"` -> `make lint` -> `git stash pop` to get linting to evaluate.

OTOH, git hooks can be configured to run only against staged changes. This totally avoid the above problems in an intuitive way.

### Sequential execution and failing early
Current `make lint` runs several commands in order. It stops on the first failure with limited debugging information.

Git hooks can safely be ran in parallel and all checks will complete (can be configured to fail fast). This allows to collect all issues with detailed diagnostics at once and fix them.

### Files desync
If you run `make lint` and start editing files while the checks run, individual checks will run on **different versions of the file** making the check results unreliable. This is because it's using the current files instead of the git tree. This prevent you from working on code while the checks run.

git hooks completely avoid this problem by using the staged versions of the files instead of the current files. It also ensures all hooks are applied on the same version of the file.

### Diagnostics
Using `make lint` will eat the error from the command that failed and only return a one line diagnosis.

pre-commit hooks print the full errors and allow you to dump logs to a file when debugging.

### Local / CI sync
`pre-commit` will run the same locally and remotely. It automatically uses isolated virtual environments and there are some mature pre-built GitHub actions available.

### More checks
Pre-commit offers a lot of useful checks out of the box that aren't typically part of formatting or linting tools. for example, check for syntax errors in .py, .toml, .yaml, etc.; validate `pyproject.toml` definition; check github actions yaml definition;

## Implementation
Commit hooks should be **fast** i.e., something you run every time you save a file. This includes formatting, linting, but excludes type checking, testing IMO.

- Create a `.pre-commit-config.yaml` to call our existing tools (`ruff`, `black`)
- Add a GitHub actions workflow that runs the `pre-commit`
- Edit `make lint` and `make format` to call utilities via `pre-commit`

The marimo repository is a good example of using `Makefile` and `pre-commit` together.

## Alternatives
- [prek](https://github.com/j178/prek) is a Rust-based alternative. It's fully backwards-compatible with pre-commit. It's generally faster and has better venv management via `uv` It's deemed "not ready for production" but is used by Apache Airflow.

Contributor guide

Open the contributing guide

Research direction

Inspect the existing Makefile targets and current linting workflow, then review how ruff and black are invoked. Add the mentioned .pre-commit-config.yaml and GitHub Actions workflow, and update make lint and make format to use the hooks. Done means staged-file checks run locally and the same pre-commit checks pass in CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, github-actions, python
Domain
ci-cd, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.