apache / apache/arrow

[Format] Add `arrow.range` canonical extension type for bounded ranges

Open
#50,027 2 comments 0 reactions 1 assignee Claimed by @Hoeze View on GitHub
Component: Format Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Arrow has no canonical way to represent a bounded range (a mathematical interval with a lower and an upper endpoint), e.g. a numeric range `[0, 10)`, a date range, or a timestamp period. Today such data is modeled ad hoc with two separate columns or with system-specific extension types, which hurts interoperability. A canonical range type will be useful to libraries like Pandas, Polars/Polars-bio, IRanges/PyRanges, database connectors, ...

Note this is distinct from Arrow's existing calendar `Interval` type (`INTERVAL_MONTHS` / `INTERVAL_DAY_TIME` / `INTERVAL_MONTH_DAY_NANO`), which represents a duration (a signed amount of time), not a bounded set. Databases like PostgreSQL make the same distinction: SQL uses `INTERVAL` for durations and `RANGE` / `PERIOD` for bounded sets. This proposal follows that convention by naming the type `arrow.range`.

## Proposed design:
- Extension name: `arrow.range`.
- Storage type: `Struct`. When subtype `T` is nullable, a null bound represents an unbounded (infinite) endpoint.
- Field names `lower` / `upper` (PostgreSQL convention) are chosen deliberately for ordering clarity. (Note that Pandas uses `left` / `right` for the field names)
- The subtype `T` may be any orderable Arrow type (the numeric, temporal and decimal families, etc.). Nested or non-comparable types are out of scope.
- Metadata: a JSON object `{"closed": "..."}`.
- Parameter `closed`: one of `left`, `right`, `both`, `neither` (pandas vocabulary; `left` = lower inclusive / upper exclusive, etc.).
- `closed` is required on the wire so a serialized `arrow.range` is always unambiguous. Unknown JSON keys are ignored for forward compatibility.

- A range is empty implicitly when `lower > upper`, or when `lower == upper` with at least one bound exclusive. A range with `lower > upper` is therefore valid (it denotes the empty set), not an error.

### Relation to pandas

This mirrors pandas' interval support and deliberately reuses its vocabulary:

- `pandas.Interval` is the scalar form: an immutable bounded interval whose `closed` parameter takes exactly `left`, `right`, `both`, or `neither`; the vocabulary adopted here for the `closed` metadata.
- `pandas.IntervalIndex` / `pandas.arrays.IntervalArray` is the columnar form: it stores parallel `left` and `right` bound arrays, directly analogous to the proposed `Struct` storage.
- Crucially, `closed` is part of pandas' dtype itself (`interval[T, left]` and `interval[T, right]` are distinct dtypes), so a typed interval column carries exactly one `closed`: constructing an array from intervals with differing closedness raises `ValueError`, and concatenating columns of differing closedness falls back to untyped `object` dtype. This uniform, type-level `closed` maps one-to-one onto the proposed object-level `closed` metadata; no per-element closedness is required.

So `arrow.range` would give pandas' `IntervalArray` / `IntervalIndex` a natural, lossless Arrow representation for round-tripping.

### Component(s)

Format

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.