pypa / pypa/setuptools

bdist_egg --exclude-source-files does not work with Python3

Open
#844 6 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Needs Triage
Dominant language
Python
Stars
2.9k
Forks
1.4k
Avg merge
1d 1h
Merged PRs (30d)
1

Description

(Coming from here: https://bugs.python.org/issue28630)

I'm trying to prepare an egg file without source code.
I had used python setup.py bdist_egg --exclude-source-files, which does not work with Python3.

After digging into the issue more, I found PEP-3147 is related to the problem.
https://www.python.org/dev/peps/pep-3147/

$ git clone https://github.com/dmiyakawa/python_greeting.git
..
$ cd python_greeting
$ python --version
Python 3.5.2
$ python setup.py bdist_egg --exclude-source-files
..
$ unzip -l dist/greeting-1.0.0-py3.5.egg
Archive:  dist/greeting-1.0.0-py3.5.egg
  Length      Date    Time    Name
---------  ---------- -----   ----
        1  11-09-2016 16:31   EGG-INFO/dependency_links.txt
       57  11-09-2016 16:31   EGG-INFO/entry_points.txt
      319  11-09-2016 16:31   EGG-INFO/PKG-INFO
      213  11-09-2016 16:31   EGG-INFO/SOURCES.txt
        9  11-09-2016 16:31   EGG-INFO/top_level.txt
        1  11-09-2016 16:31   EGG-INFO/zip-safe
      139  11-09-2016 16:31   greeting/__pycache__/__init__.cpython-35.pyc
      392  11-09-2016 16:31   greeting/__pycache__/greeting.cpython-35.pyc
---------                     -------
     1131                     8 files

Here, greeting/__pycache__/__init__.cpython-35.pyc and greeting/__pycache__/greeting.cpython-35.pyc are not where they should be.
See the following results (with Python 2.7.12)

$ python --version
Python 2.7.12
$ python setup.py bdist_egg --exclude-source-files
..
$ unzip -l dist/greeting-1.0.0-py2.7.egg
Archive:  dist/greeting-1.0.0-py2.7.egg
  Length      Date    Time    Name
---------  ---------- -----   ----
        1  11-09-2016 16:32   EGG-INFO/dependency_links.txt
       57  11-09-2016 16:32   EGG-INFO/entry_points.txt
      319  11-09-2016 16:32   EGG-INFO/PKG-INFO
      213  11-09-2016 16:32   EGG-INFO/SOURCES.txt
        9  11-09-2016 16:32   EGG-INFO/top_level.txt
        1  11-09-2016 16:32   EGG-INFO/zip-safe
      143  11-09-2016 16:32   greeting/__init__.pyc
      482  11-09-2016 16:32   greeting/greeting.pyc
---------                     -------
     1225                     8 files

According to the chart in PEP-3147, original py file must also exist to make pyc cache in __pycache__ effective.

To let bdist_egg's --exclude-sourcce-files option work as it worked in Python2 era, pyc files in __pycache__ must be at where legacy Python2 placed. I tried tweaking the Py3's egg content with the following code, and the tweaked egg worked nicely.

RE_PYCACHE_FILE = re.compile('(.+/)__pycache__/(.+)\.cpython-3\d(.pyc)$')

with tempfile.TemporaryDirectory() as tmpdir:
    names_lst = []
    tmpdir_abs_path = os.path.abspath(tmpdir)
    with ZipFile(egg_path, 'r') as zf:
        for name in zf.namelist():
            m = RE_PYCACHE_FILE.match(name)
            if m:
                w_name = m.group(1) + m.group(2) + m.group(3)
            else:
                w_name = name
            w_path = os.path.join(tmpdir_abs_path, w_name)
            dir_path = os.path.dirname(w_path)
            if not os.path.exists(dir_path):
                os.makedirs(dir_path)
            with open(w_path, 'wb') as file_to_write:
                with zf.open(name) as file_to_read:
                    file_to_write.write(file_to_read.read())
            names_lst.append(w_name)
    with ZipFile(egg_path, 'w', compression=ZIP_DEFLATED) as zf:
        for name in names_lst:
            r_path = os.path.join(tmpdir_abs_path, name)
            zf.write(r_path, arcname=name)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the bdist_egg entry point and reproduce the issue with python setup.py bdist_egg --exclude-source-files using the Python 3 example. Compare the generated archive with the Python 2 listing and review PEP 3147's cache-path rules. Done means Python 3 eggs exclude source files while placing bytecode where they remain usable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.