python / python/cpython

Fix forced arg format in AC-processed modules with custom converters

Open
#94,512 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-argument-clinic type-feature
Dominant language
Python
Stars
77.2k
Forks
36k
PR merge metrics
PR metrics pending

Description

There are custom Argument Clinic converters that define format_unit but omit parse_arg. As a result, generation of positional argument parsers is forced to back up from the fastest possible _PyArg_CheckPositional to slower _PyArg_ParseStack-based format strings.

Here is a list of such classes (and fixing PRs except complex cases):

  • Modules\_multiprocessing\multiprocessing.c (gh-94517)

    • HANDLE_converter
  • Modules\_multiprocessing\semaphore.c

    • SEM_HANDLE_converter
  • Modules\overlapped.c (gh-94516)

    • OVERLAPPED_converter
    • HANDLE_converter
    • ULONG_PTR_converter
    • DWORD_converter
    • BOOL_converter
  • Modules\posixmodule.c (gh-122516)

    • pid_t_converter
    • idtype_t_converter
    • id_t_converter
    • intptr_t_converter
    • Py_off_t_converter
  • Modules\resource.c (gh-94515)

    • pid_t_converter
  • PC\msvcrtmodule.c (gh-94514)

    • HANDLE_converter
  • PC\winreg.c (gh-94513)

    • REGSAM_converter
    • DWORD_converter
    • HKEY_converter

An example of such a converter:

class BOOL_converter(CConverter):
    type = 'BOOL'
    format_unit = 'i'

class pid_t_converter(CConverter):
    type = 'pid_t'
    format_unit = '" _Py_PARSE_PID "'

I'm going to teach all of them about low-level generation by replacing manual format_unit definitions with:

  • inheritance from a corresponding builtin converter where possible
  • and custom parse_args in other places.

For the example it gives:

class BOOL_converter(int_converter):
    type = 'BOOL'

class pid_t_converter(CConverter):
    type = 'pid_t'
    # Left as a backup for potential complex cases
    format_unit = '" _Py_PARSE_PID "'

    def parse_arg(self, argname, displayname):
        return """
            {paramname} = PyLong_AsPid({argname});
            if ({paramname} == -1 && PyErr_Occurred()) {{{{
                goto exit;
            }}}}
            """.format(argname=argname, paramname=self.parser_name)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Modules/_multiprocessing/semaphore.c and its SEM_HANDLE_converter, then read the listed converter examples and the corresponding checked-off modules for comparison. Review how Argument Clinic uses format_unit and parse_arg during positional parser generation. Done means the converter no longer forces the slower format-string parser while retaining correct argument parsing.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
performance, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.