apache / apache/beam

[Feature Request]: Add library stubs or py.typed marker

Open
#27,906 5 comments 4 reactions 0 assignees View on GitHub
new feature P2 python stale
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

### What would you like to happen?

Support running `mypy` on my own Beam pipeline code by adding type stubs.

Simply importing `apache_beam` fails with `mypy`.

```py
#-- main.py
import apache_beam as beam
```

Install and run `mypy`.

```sh
pip install mypy apache-beam

mypy main.py
```

Gives the following output:

```
main.py:1: error: Skipping analyzing "apache_beam": module is installed, but missing library stubs or py.typed marker [import]
main.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)
```

They can be [automatically generated with `stubgen`](https://mypy.readthedocs.io/en/stable/stubgen.html), it would be nice to make that part of the release process on each new version. Since Beam already has type hints, I don't think there is too much tweaking needed.

I did try to generate them, but it failed trying to parse some complex type hint.

### Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

### Issue Components

- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with main.py, apache_beam, and mypy, then inspect the Python SDK release process and the stubgen path mentioned in the issue. Determine how the existing type hints and the complex hint that blocked generation should be handled; done means a released apache_beam package can be analyzed by mypy without the missing-stubs or py.typed error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
developer-experience, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.