numpy / numpy/numpy

ENH: Add templates for `f2py` c functions and handle more C types

Open
#25,229 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

00 - Bug 01 - Enhancement component: numpy.f2py
Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

Consider the following (after the corrected handling of ISO_C in #25226):

   Ignoring map {'integer':{'c_short':'short int'}}: 'short int' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_long':'long int'}}: 'long int' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_long_long':'long long int'}}: 'long long int' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_signed_char':'signed char'}}: 'signed char' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_size_t':'size_t'}}: 'size_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_int8_t':'int8_t'}}: 'int8_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_int16_t':'int16_t'}}: 'int16_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_int32_t':'int32_t'}}: 'int32_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_int64_t':'int64_t'}}: 'int64_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_int_least8_t':'int_least8_t'}}: 'int_least8_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_int_least16_t':'int_least16_t'}}: 'int_least16_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_int_least32_t':'int_least32_t'}}: 'int_least32_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_int_least64_t':'int_least64_t'}}: 'int_least64_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_int_fast8_t':'int_fast8_t'}}: 'int_fast8_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_int_fast16_t':'int_fast16_t'}}: 'int_fast16_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_int_fast32_t':'int_fast32_t'}}: 'int_fast32_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_int_fast64_t':'int_fast64_t'}}: 'int_fast64_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_intmax_t':'intmax_t'}}: 'intmax_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_intptr_t':'intptr_t'}}: 'intptr_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'integer':{'c_ptrdiff_t':'intptr_t'}}: 'intptr_t' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Mapping "real(kind=c_float)" to "float"
	Mapping "real(kind=c_double)" to "double"
	Ignoring map {'real':{'c_long_double':'long double'}}: 'long double' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'complex':{'c_float_complex':'float _Complex'}}: 'float _Complex' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'complex':{'c_double_complex':'double _Complex'}}: 'double _Complex' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'complex':{'c_long_double_complex':'long double _Complex'}}: 'long double _Complex' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Ignoring map {'logical':{'c_bool':'_Bool'}}: '_Bool' must be in ['double', 'float', 'long_double', 'char', 'signed_char', 'unsigned_char', 'short', 'unsigned_short', 'int', 'long', 'long_long', 'unsigned', 'complex_float', 'complex_double', 'complex_long_double', 'string', 'character']
	Mapping "character(kind=c_char)" to "char"

Which can be seen by passing verbose = True to process_f2cmap_dict while mapping the iso_c values in capi_maps.py. To silence these, there are the (commented out) entries in iso_c2py_map in _isocbind.py.

The reason they are commented out is, otherwise, F2PY will internally construct and try to use "construction" functions like these:

needs['short_from_pyobj'] = ['int_from_pyobj']
cfuncs['short_from_pyobj'] = """
static int
short_from_pyobj(short* v, PyObject *obj, const char *errmess) {
    int i = 0;
    if (int_from_pyobj(&i, obj, errmess)) {
        *v = (short)i;
        return 1;
    }
    return 0;
}

Which are defined in cfuncs.py. Now most of these (almost all) follow a pattern, they just cast and call int_from_pyobj or similar. However, for the iso_c (and really in general for the remaining C types) there should be similar functions.

Solution

This could be copy pasted, but really they should be generated on the fly out of templates like the meson backend and its meson.build backend. That would also finally allow for syntax highlighting, currently modifying the cfuncs.py C code is rather painful.

Addenum

For #25226 the mapping of types isn't really required. Here are the changes needed to _isocbind.py to reproduce the issue:

iso_c_binding_map = {
    'integer': {
        'c_int': 'int',
        'c_short': 'short int',
        'c_long': 'long int',
        'c_long_long': 'long long int',
        'c_signed_char': 'signed char',
        'c_size_t': 'size_t',
        'c_int8_t': 'int8_t',
        'c_int16_t': 'int16_t',
        'c_int32_t': 'int32_t',
        'c_int64_t': 'int64_t',
        'c_int_least8_t': 'int_least8_t',
        'c_int_least16_t': 'int_least16_t',
        'c_int_least32_t': 'int_least32_t',
        'c_int_least64_t': 'int_least64_t',
        'c_int_fast8_t': 'int_fast8_t',
        'c_int_fast16_t': 'int_fast16_t',
        'c_int_fast32_t': 'int_fast32_t',
        'c_int_fast64_t': 'int_fast64_t',
        'c_intmax_t': 'intmax_t',
        'c_intptr_t': 'intptr_t',
        'c_ptrdiff_t': 'intptr_t',
    },
    'real': {
        'c_float': 'float',
        'c_double': 'double',
        'c_long_double': 'long double'
    },
    'complex': {
        'c_float_complex': 'float _Complex',
        'c_double_complex': 'double _Complex',
        'c_long_double_complex': 'long double _Complex'
    },
    'logical': {
        'c_bool': '_Bool'
    },
    'character': {
        'c_char': 'char'
    }
}

# TODO: At some point these should be included, but then they'd need special
# handling in cfuncs.py e.g. needs[int64_t_from_pyobj] These are not very hard
# to add, since they all derive from the base `int_from_pyobj`, e.g. the way
# `short_from_pyobj` and others do

isoc_c2pycode_map = {
    'int': 'i',  # int
    'short int': 'h',  # short int
    'long': 'l',  # long int
    'long long': 'q',  # long long int
    'signed char': 'b',  # signed char
    'size_t': 'I',  # size_t (approx unsigned int)
    'int8_t': 'b',  # int8_t
    'int16_t': 'h',  # int16_t
    'int32_t': 'i',  # int32_t
    'int64_t': 'q',  # int64_t
    'int_least8_t': 'b',  # int_least8_t
    'int_least16_t': 'h',  # int_least16_t
    'int_least32_t': 'i',  # int_least32_t
    'int_least64_t': 'q',  # int_least64_t
    'int_fast8_t': 'b',  # int_fast8_t
    'int_fast16_t': 'h',  # int_fast16_t
    'int_fast32_t': 'i',  # int_fast32_t
    'int_fast64_t': 'q',  # int_fast64_t
    'intmax_t': 'q',  # intmax_t (approx long long)
    'intptr_t': 'q',  # intptr_t (approx long long)
    'ptrdiff_t': 'q',  # intptr_t (approx long long)
    'float': 'f',  # float
    'double': 'd',  # double
    'long double': 'g',  # long double
    'float _Complex': 'F',  # float  _Complex
    'double _Complex': 'D',  # double  _Complex
    'long double _Complex': 'D',  # very approximate complex
    '_Bool': 'i',  #  Bool but not really
    'char': 'c',   # char
}

iso_c2py_map = {
    'int': 'int',
    'short int': 'int',                 # forced casting
    'long': 'int',
    'long long': 'long',
    'signed char': 'int',           # forced casting
    'size_t': 'int',                # approx Python int
    'int8_t': 'int',                # forced casting
    'int16_t': 'int',               # forced casting
    'int32_t': 'int',
    'int64_t': 'long',
    'int_least8_t': 'int',          # forced casting
    'int_least16_t': 'int',         # forced casting
    'int_least32_t': 'int',
    'int_least64_t': 'long',
    'int_fast8_t': 'int',           # forced casting
    'int_fast16_t': 'int',          # forced casting
    'int_fast32_t': 'int',
    'int_fast64_t': 'long',
    'intmax_t': 'long',
    'intptr_t': 'long',
    'ptrdiff_t': 'long',
    'float': 'float',
    'double': 'float',              # forced casting
    'long double': 'float',         # forced casting
    'float _Complex': 'complex',     # forced casting
    'double _Complex': 'complex',
    'long double _Complex': 'complex', # forced casting
    '_Bool': 'bool',
    'char': 'bytes',                  # approx Python bytes
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with cfuncs.py, _isocbind.py, and capi_maps.py, especially process_f2cmap_dict and the commented iso_c2py_map entries. Compare the existing meson backend and meson.build templates to understand the proposed generation approach. Done means the remaining ISO_C and C types are represented by generated construction functions without the reported mapping warnings.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, fortran, python
Domain
compilers, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.