[C++][Python] Allow an ExtensionType to register or implement custom casts
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
With ARROW-14500 and ARROW-15545 (https://github.com/apache/arrow/pull/14106), we are allowing to cast "storage_type" -> "extension" (and the cast the other way around already worked as well).
Initially, that PR allowed any cast from "any" -> "extension", as long as the input type could be cast to the storage type (so deferring to the "any" -> "storage_type" cast). However, because whether a certain cast makes sense or not depends on the semantics of the extension type, it was restricted to exactly matching storage_type.
One idea could be to still allow the other casts behind a cast option flag, like `allow_non_storage_extension_casts` (or a better name), so the user can explicitly allow to cast to/from any type (as long as the cast from/to the storage type works).
That could help for the user, but for certain casts, the ExtensionType might also want to control _how_ such a cast is done. For example, for casting to/from string type (which would be useful for reading/writing CSV files, or for repr), you typically will want to do something different than casting your storage array to string.
A more general solution could thus be to have a mechanism for the ExtensionType to implement a certain cast kernel itself, and register this to the C++ cast dispatching.
**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-17890) / @jorisvandenbossche
**Note**: *This issue was originally created as [ARROW-17890](https://issues.apache.org/jira/browse/ARROW-17890). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reading ARROW-14500, ARROW-15545, and pull request 14106 to understand the existing storage_type and extension casting behavior. Then compare the option-flag idea with the more general custom cast-kernel mechanism; done means an ExtensionType can control the relevant casts and the behavior is covered by the project's existing cast dispatching tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100