apache / apache/devlake

[Proposal] Deprecate Python plugin framework and consolidate on Go

Open
#9,092 2 comments 5 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
3.1k
Forks
808
Avg merge
1d 8h
Merged PRs (30d)
49

Description

## Proposal: Deprecate the Python plugin framework and consolidate on Go

### Summary

Propose to deprecate the Python plugin framework (`backend/python/`, a.k.a. PyDevLake) and remove it from the codebase after a 3-month deprecation window. Going forward, all plugins should be written in Go. This reduces maintenance overhead, simplifies the Docker image and CI, and focuses contributor effort on a single, well-supported runtime.

### Background

The Python plugin framework was introduced in Feb 2023 (#3702, #4438) to lower the barrier for writing plugins. In practice it never saw adoption:

- Only **one** plugin ever shipped on it: `azuredevops` (Python).
- The `dbt` Python plugin was already removed on 2026-08-31 (#8970).
- A fully-featured Go implementation, `azuredevops_go`, now exists and is actively developed (on-premises support landed 2026-08-16, #9014).

### Evidence: Python plugins are not being maintained

| Plugin | Last meaningful commit | Note |
| --- | --- | --- |
| `azuredevops` (Python) | 2025-12-31 `feat: add environment_pattern` (#8671) | No feature/fix since; only deps refresh & runtime bumps |
| `pydevlake` framework | 2025-12-31 (same as above) | Last real fix: #8957 on 2026-07-15 (missing composite PKs) |
| `dbt` (Python) | removed 2026-08-31 (#8970) | Already gone |

In the last ~12 months, the only commits touching `backend/python/` are dependency refreshes, Poetry/uv lock bumps, a Python 3.9→3.11 runtime bump, and a single primary-key fix. Meanwhile the Go side shipped 40+ plugins and continues to receive features weekly.

### Benefits

1. **One runtime to maintain.** Eliminate the parallel PyDevLake framework, its RPC bridge, `pydevlake` package, `uv`/Poetry tooling, and Python-specific test/build scripts.
2. **Simpler Docker build.** Today `backend/Dockerfile` is based on `python:3.11-slim-bookworm`, installs python3-dev/pip/uv, copies `python/`, and runs `build.sh`. After removal the image can be a slim Go-based image — smaller, fewer CVEs, faster builds. `Dockerfile.local`'s "empty python plugins dir" workaround also goes away.
3. **Simpler CI.** Two workflows (`.github/workflows/test.yml`, `test-e2e.yml`) and Make targets (`python-dep`, `python-unit-test`, `build-python`, `build-pydevlake`) can be dropped.
4. **Clearer contributor story.** New plugin authors get one path (Go) with 40+ reference implementations instead of choosing between two frameworks where one is unmaintained.
5. **No functional loss.** The only remaining Python plugin (`azuredevops`) is superseded by `azuredevops_go`, which already has broader feature coverage (Server/on-prem support).

### Plan & Timeline

| Phase | Window | Actions |
| --- | --- | --- |
| **1. Warning** | T+0 (on issue merge) | Add deprecation notices to: `backend/python/README.md`, plugin `README.md`s, server startup log line, and `config-ui` plugin selection UI. Mark `azuredevops` (Python) as deprecated in the plugin registry; point users to `azuredevops_go`. |
| **2. Freeze** | T+1 month | Stop accepting new features/fixes for Python plugins. Only critical security fixes land. Add a CI gate rejecting new files under `backend/python/plugins/`. |
| **3. Migration assist** | T+1~T+3 months | Ship a migration guide + script to move any remaining `azuredevops` (Python) connections/scopes to `azuredevops_go`. Document data-layer equivalence. |
| **4. Removal** | T+3 months | Delete `backend/python/`, PyDevLake, the Python Dockerfile stage, `make python-*` targets, and Python CI jobs. Remove `azuredevops` (Python) from the plugin registry. |

Total deprecation window: **3 months** from the date this proposal is accepted.

### Risks & Mitigations

- **Existing users of `azuredevops` (Python):** Mitigated by `azuredevops_go` parity + migration guide in Phase 3. The Go plugin already covers cloud + on-prem.
- **External contributors with in-progress Python plugins:** The 3-month window + the warning phase gives them time to port. No private Python plugins are known to exist in the wild.
- **Reverting later:** Not needed; Go covers all current and planned use cases.

### What we need

- 👍 / 👎 from maintainers and community on deprecating Python.
- Owners to confirm `azuredevops_go` feature parity with the Python `azuredevops` plugin (esp. CI/CD environment extraction).
- Volunteers to write the Phase-3 migration guide.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing backend/python/, backend/Dockerfile, Dockerfile.local, .github/workflows/test.yml, .github/workflows/test-e2e.yml, and the listed Make targets. Then compare the Python azuredevops plugin with azuredevops_go, especially CI/CD environment extraction. Done means maintainers agree on the proposal and owners are identified for the warning, migration, and removal phases.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, go, python
Domain
backend, build-system, ci-cd, developer-experience, devops
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.