con / con/git-annex

Build Python wheels for git-annex and upload them to PyPi

Open
#223 9 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
29
Forks
5
PR merge metrics
No merged PRs in 30d

Description

I propose to build Python wheels for git-annex...

- to be able to declare proper "native" dependencies on it
- to make it accessible to venv management (e.g. by `uv`) with download/installation caching
- to spare users the need to use the custom `datalad-installer` to get git-annex deploy on the more challenging platforms

PyPi has a 60MB package limit. git-annex binaries are ~30MB (compressed). This would fit.

I include a patch to the git-annex sources below that enabled running

```
hatch build -t wheel
```

in a git-annex source tree that has been built via

```
stack setup
stack build --only-dependencies --extra-include-dirs=$PWD --extra-lib-dirs=$PWD
stack install --no-haddock --local-bin-path bin
```

to produce something like `git_annex-10.20250417.dev28+g7bf6e1df8a-py3-none-linux_x86_64.whl`. The stack parts of the build could be integrated more. However, I am proposing this draft already now, because some issues are unresolved and additional ideas are welcome. Those are:

`git-annex-shell`, `git-remote-annex`, and `git-remote-tor-annex` are not installed. These are merely symlinks to the git-annex binary, but: A python wheel is a ZIP file, and it does not support symlinks.

Worse, if I do not give a damn and include copies instead of symlinks, the wheel size multiplies 🙄

I assume that git-annex looks at argv[0] to decide what it should do, so adding a wrapper script is not really an option. I further assume that the git-annex installer for windows just creates these copies on the target system.

The wheel specification has no concept of a post-install step.

What could be done?

Here is the patch:

```diff
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000000..f6888ad849
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,31 @@
+[project]
+name = "git-annex"
+dynamic = ["version"]
+description = "Add your description here"
+readme = {file = "README", content-type = "text/markdown"}
+requires-python = ">=3.9"
+
+[build-system]
+requires = [
+ "hatchling",
+ "hatch-vcs",
+]
+build-backend = "hatchling.build"
+
+[tool.hatch.version]
+source = "vcs"
+
+[tool.hatch.build.targets.wheel.shared-data]
+"bin/git-annex" = "bin/git-annex"
+
+[tool.hatch.build.targets.wheel]
+# we need to ship at least one file
+only-include = ["python/py.typed"]
+
+[tool.hatch.build.targets.wheel.sources]
+# give the 'only-include' files a base directory in the wheel
+"python" = "git-annex"
+
+[tool.hatch.build.targets.wheel.hooks.custom]
+# custom build hook to set some metadata
+path = "python/build_hook_plugin.py"
diff --git a/python/build_hook_plugin.py b/python/build_hook_plugin.py
new file mode 100644
index 0000000000..02f556a4a1
--- /dev/null
+++ b/python/build_hook_plugin.py
@@ -0,0 +1,18 @@
+from sysconfig import get_platform
+from typing import Any
+
+from hatchling.builders.hooks.plugin.interface import BuildHookInterface
+
+
+class SpecialBuildHook(BuildHookInterface):
+
+ def initialize(
+ self,
+ version: str, # noqa: ARG002
+ build_data: dict[str, Any],
+ ) -> None:
+ # we have platform-specific builds
+ build_data['pure_python'] = False
+ # set a tag that says: any python3 for the build platform
+ build_data['tag'] = \
+ f'py3-none-{get_platform().replace("-", "_").replace(".", "_")}'
diff --git a/python/py.typed b/python/py.typed
new file mode 100644
index 0000000000..e69de29bb2
```

It goes without saying that more metadata could and should be added. Importantly, this patch already does dynamic versioning using the git-annex repository (tags).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the proposed pyproject.toml and python/build_hook_plugin.py, then reproduce the documented stack setup, stack build, stack install, and hatch build commands. Review how the wheel handles git-annex-shell, git-remote-annex, and git-remote-tor-annex, along with package metadata and the PyPI size limit. Done means a platform-specific wheel builds and installs the required commands without unresolved packaging issues.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, release
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.