pypa / pypa/setuptools

Generated files must be present before invocation to be included in package_data

Open
#1,064 4 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Needs Triage
Dominant language
Python
Stars
2.9k
Forks
1.4k
Avg merge
1d 1h
Merged PRs (30d)
1

Description

I've created the following repository to demonstrate the problem:
https://github.com/JonathonReinhart/setuptools-package_data-issue

Copying what I've shown there:

This repository is an MCVE for the following issues:

Problem statement

When python setup.py sdist bdist_wheel is invoked from a clean starting point, data files (identified in package_data), which are generated during setup.py hooks, will not be included in the wheel.

  • From a clean starting point (git clean -fdx) (primarily removes mkpkg/generated_data):
    • python setup.py bdist_wheel - OK
    • python setup.py sdist then python setup.py bdist_wheel - OK
    • python setup.py sdist bdist_wheel - BROKEN

Steps to reproduce

First:

  • Run git clean -fdx -- remove any generated files
  • Run python setup.py sdist bdist_wheel
  • Observe only the following line in the output:
    • copying build/lib/mypkg/__init__.py -> build/bdist.linux-x86_64/wheel/mypkg
  • Run unzip -l dist/mypkg-1.2.3-py2-none-any.whl
    • Observe that mypkg/generated_data is missing from the wheel

Now that mypkg/generated_data is present:

  • Run python setup.py sdist bdist_wheel again
  • Observe the following lines in the output:
    • copying build/lib/mypkg/generated_data -> build/bdist.linux-x86_64/wheel/mypkg
    • copying build/lib/mypkg/__init__.py -> build/bdist.linux-x86_64/wheel/mypkg
  • Run unzip -l dist/mypkg-1.2.3-py2-none-any.whl
    • Observe that mypkg/generated_data is now present in the wheel

Clean the directory, and run other variations of sdist and bdist_wheel (as mentioned above) and observe that the problem does not manifest.

Workarounds that do not work

  • Hooking build_py instead of build

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with git clean -fdx and python setup.py sdist bdist_wheel, then inspect setup.py, its generation hooks, package_data, sdist, and bdist_wheel behavior. Compare the combined command with the separate sdist and bdist_wheel runs. Done means generated_data is included in the wheel from a clean starting point.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.