packagecontrol / packagecontrol/thecrawl

What's been cooking

Open
#343 2 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

crawler
Dominant language
Python
Stars
19
Forks
7
Avg merge
4h 53m
Merged PRs (30d)
14

Description

0.15.0

  • Library definitions are easier to read, match, and debug

Library entries can be tricky on both sides: release definitions in the
registry, and resolved release JSON in the workspace. So I added tooling
around that.

crawl_libraries.py --name still crawls one library in dry-run mode but prints a
platform/Python matrix now instead of raw JSON. Like so:

$ uv run -m scripts.crawl_libraries --name numpy
numpy release matrix; -v to see the raw JSON output
Source: pypi:cache
Latest version: 2.4.6

                py33  py38  py313  py314
-------------+--------------------------
windows-x64     X     A     B      B
osx-x64         A'    A     B      B
osx-arm64       -     A     B      B
linux-x64       A''   A     B      B
linux-arm64     -     A     B      B

A   = 1.24.4
A'  = 1.11.0
A'' = 1.10.4
B   = 2.4.6
X   = no version found, run -v for details

For the definition side of the problem, --try can be used to try out release definitions
without writing a registry.json. It accepts JSON, a small key: value shorthand,
or stdin. It can be combined with either --name or --explain. E.g.

Use --name to see which version the crawler selects for a given definition.

$ uv run -m scripts.crawl_libraries --name numpy --try <<'DEF'
base: pypi:numpy
platforms: linux-x64
python_versions: 3.14
DEF
numpy release matrix; -v to see the raw JSON output
Source: pypi:cache
Latest version: 2.4.6

              py314
-----------+-------
linux-x64     A

A = 2.4.6

--explain then shows how that shorthand expands into the normalized
release definitions the crawler actually matches against.

$ uv run -m scripts.crawl_libraries --explain numpy --try <<'DEF'
base: pypi:numpy
platforms: linux-x64
python_versions: 3.14
DEF
{
  "author": "Numpy",
  "description": "NumPy is the fundamental package for scientific computing with Python.",
  "issues": "https://github.com/numpy/numpy/issues",
  "name": "numpy",
  "schema_version": "4.0.0",
  "source": "https://raw.githubusercontent.com/packagecontrol/channel/refs/heads/main/repository.json"
}

#   Input definition                 Normalized variation
───────────────────────────────────────────────────────────────────────────────────────────
1   {                                {
      "base": "pypi:numpy",            "asset": [
      "platforms": "linux-x64",          "*-${version}-cp314-cp314m-manylinux*_x86_64.whl",
      "python_versions": "3.14"          "*-${version}-cp314-cp314-manylinux*_x86_64.whl",
    }                                    "*-${version}-py3-none-manylinux*_x86_64.whl",
                                         "*-${version}-py2.py3-none-manylinux*_x86_64.whl",
                                         "*-${version}-py3-none-any.whl",
                                         "*-${version}-py2.py3-none-any.whl"
                                       ],
                                       "base": "https://pypi.org/project/numpy",
                                       "platform": "linux-x64",
                                       "python_version": "3.14",
                                       "sublime_text": "*",
                                       "tag_prefix": "v?",
                                       "version": "*"
                                     }

0.15.6

Iterates on the --try mode. You can now emit the name and it will find out.

$ uv run -m scripts.crawl_libraries --try "base: pypi:lxml"

You can also use ";" to separate properties, e.g.

$ uv run -m scripts.crawl_libraries --try "base: pypi:pyobjc-framework-Cocoa; platform: osx;"
pyobjc-framework-Cocoa release matrix; -v to see the raw JSON output
Source: pypi:cache
Latest version: 12.2

              py38  py314
-----------+-------------
osx-x64       A     B
osx-arm64     A     B

A = 10.3.2
B = 12.2

This also works with --explain

$ uv run -m scripts.crawl_libraries --try "base: pypi:pyobjc-framework-Cocoa; platform: osx;" --explain
$ uv run -m scripts.crawl_libraries --try "base: pypi:pyobjc-framework-Cocoa; platform: osx;" --explain --json

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The message describes scripts.crawl_libraries and its --name, --try, and --explain entry points, but does not identify a requested change. Read scripts.crawl_libraries first and run the documented uv commands to understand the current behavior; no completion criteria are stated in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, tooling
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.