digitize is not the inverse of histogram (right limit treated differently)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
I think digitize should change its behavior with respect to binning a value that matches the rightmost bin's right edge. Such a value should go in the rightmost bin and not be classified as "beyond the bounds". This would make the output of histogram compatible to digitize (currently it's not).
Example (short explanation)
>>> from numpy import histogram
>>> from numpy import digitize
>>> data = [1, 2, 3, 4, 5, 6, 7]
>>> histvals, binedges = histogram(data, 3)
>>> binedges
array([ 1., 3., 5., 7.])
>>> histvals
array([2, 2, 3])
# Up to here, a histogram has been created characterized by
# four bin edges (three bins). The rightmost bin contains
# three data values, as its right edge is considered to belong
# to the bin.
# Now assign the data values to given bins
# (represented by `binedges`) via `digitize`
>>> digitize(data, binedges)
array([1, 1, 2, 2, 3, 3, 4])
# The first two data values have been assigned to bin 1,
# the next two data values were assigned to bin 2,
# the next two data values were assigned to bin 3,
# the remaining data value has been declared as "out of bounds".
# This is where `histogram` and `digitize` behave differently by
# default.
# Reproduce `histogram` binning by manually shifting the
# rightmost bin edge by an epsilon value:
>>> binedges[-1] += 10**-6
>>> digitize(data, binedges)
array([1, 1, 2, 2, 3, 3, 3])
Longish explanation
I am assuming that digitize generally is considered to do the inverse operation of histogram, i.e. whereas histogram creates bins (the edges of the bins) and assigns values to bins, digitize (quote from digitize docs) "Returns the indices of the bins to which each value in input array belongs." (given the data values and the bin edges). This "reverse indexing" behavior has also been referred to in this issue: https://github.com/numpy/numpy/issues/990. It seems like other numerical frameworks provide this functionality, too.
In the histogram specs (http://docs.scipy.org/doc/numpy/reference/generated/numpy.histogram.html), the meaning of the bin edges is clarified. By default, bins contain the left edge and do not contain the right edge, except for the rightmost bin, whose right edge belongs to the bin ("The last bin, however, is [3, 4], which includes 4.")
In contrast, the digitize specs state that "Each index i returned is such that bins[i-1] <= x < bins[i]". There is no comment on the rightmost boundary, so this general statement also applies to it.
In conclusion, whereas histogram always creates the rightmost bin edge in a way that it corresponds to the maximum data value, digitize does not count this data value as part of the histogram represented by the bin edges array.
Restoring compatibility between both functions requires manually incrementing the rightmost bin edge by an epsilon value.
I think it comes down to the question of what people expect digitize to do exactly. I would also be fine with always manually correcting one boundary, but am pretty sure that that has not been the original idea behind digitize.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the digitize and histogram documentation and reproducing the reported example with the given data and bin edges. Trace the existing boundary semantics for both functions, then review related discussion before deciding whether their behavior should align. Done means an agreed boundary rule, updated tests for the rightmost edge, and consistent documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100