numpy / numpy/numpy

User-defined dtype: Can be confusing to know which conversions to provide for compatibility with core methods

Open
#10,904 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

04 - Documentation component: documentation component: numpy.dtype
Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

Relates #8952, #8860
Same campaign as #10721, #10897

When defining user-defined dtypes (using the C API), it can sometimes be a tad confusing to know which conversion to provide to ensure the underlying machinery works properly.

For example, the implementation of np.ones relies on on the following operation:

multiarray.copyto(a, 1, casting='unsafe')

When debugging, it was a tad confusing to know whether to define a conversion for np.int64 or np.int or something else. (I had gotten distracted by https://github.com/pybind/pybind11/issues/1328 - still need to finish out the related PR and test it out.)

Additionally, for identity values for reduction and such, it's also confusing.

I found that np.trace was also a tad sticky; after defining conversions for double -> my custom dtype, np.trace would still fail to find a conversion, even though I had an override defined for np.add for my custom dtype. After digging, it seems that the assign_identity_* functions relied on bool casting (PyUFunc_GenericReduction -> ... -> PyUFunc_ReduceWrapper -> assign_reduce_identity_zero -> ... -> get_cast_transfer_function).
Post-#8952, it seems that now the proper conversion to define would be (int -> Custom).
(To clarify, I'm not knocking the fix, it does seem like an improvement! I have no issue with defining conversions from both bool and int.)
(As a side note, also relates #9351 in < v1.14.0: since no overrides were found, it looped.)

That being said, it would be nice if there were hints in the documentation somewhere for compatibility with core NumPy functions; that, or at least some comments in test_rational.c.src explaining some of these heuristics?

\cc @eric-wieser @njsmith

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the user-defined dtype discussion and the conversion and identity behavior described in this issue, then inspect test_rational.c.src for the existing explanatory comments. Determine where documentation or comments should explain the conversions needed by core methods such as np.ones, reductions, and np.trace. Done means the relevant guidance is recorded clearly for dtype authors.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.