JuliaPy / JuliaPy/PythonCall.jl

new prioritisation scheme for pyconvert rules

Aperta
#661 1 commento 0 reazioni 1 assegnatario Rivendicata da @cjdoris Vedi su GitHub
Lingua principale
Julia
Stelle
1.1k
Fork
86
Merge medio
1g 22h
PR unite (30g)
3

Descrizione

## Current status

Currently a pyconvert rule consists of:
- the source python type `t`, which the rule can convert from;
- the target julia type, `T`, which the rule can convert to;
- the `priority` of the rule; and
- the function `func` implementing the rule.

When `pyconvert(R, x)` runs, it first filters the list of rules according to `t` and `T` (roughly `pyisinstance(x, t)` and `typeintersect(R, T) != Union{}`). The rules are then ordered first by `priority`, then by the specificity of `t`, then by the order the rules were defined.

The priorities are:
- `jlwrap`: for wrapped julia objects by just unwrapping them;
- `array`: for array-like objects (buffers, numpy arrays, ...);
- `canonical`: for the canonical conversion for a type, e.g. `float` to `Float64`;
- `normal`: for all other reasonable conversions.

The priorities are a bit of a hack, to work around the fact that ordering by specificity of `t` isn't quite right. For example, we *always* want to convert julia objects by unwrapping them first, so we need their rules to come first, even if the object also happens to be a `Mapping` and we are converting to `Dict`, we don't want to use the generic `Mapping` to `Dict` rule. And if the object is array-like, we want to convert by getting at the underlying memory instead of using the generic `Sequence` to `Array` rule.

## The proposal

So my proposal is to remove `priority` and add:
- the scope julia type `S`, which must be a supertype of `T`.

We further filter rules by `S` (`R <: S`, except if `R isa Union` then just one component has to match).

For ordering rules, we no longer order by priority, just by specificity of `t` and insertion order.

We also ignore `type(x).mro()` and only use strict specificity (`issubclass(t1, t2)`). That is, rules form a DAG with this partial ordering, which we flatten using insertion order to break ties.

You are only allowed to create rules where you "own" either `t` or `S`.

## Discussion

This means you can only have `S=Any` if you own `t`. Can think of `S=Any` as being `canonical` priority or higher. PythonCall will continue to "own" the Python standard library, and most rules in PythonCall will have `S=Any`. The exception is for some things currently in the `normal` priority. For example we convert `None` to `Nothing` canonically but can also go to `Missing`. In the new system, the rules will have `T=Nothing, S=Any` and `T=S=Missing`, so you generically get `Nothing` but can get `Missing` if you ask for it. Similarly `tuple` canonically converts to `Tuple` but can also go to `Array`, the rules for which will become `T=Tuple, S=Any` and `T=Array S=AbstractArray`, so you will get an `Array` if you specify `Array` or `AbstractArray`.

If you don't own `t`, then you must own `S`. This lets you define e.g. a generic conversion rule for `list` to some new `MyArray` you invented. But you can only use the rule if you specify `pyconvert(MyArray, x)`. Doing `pyconvert(AbstractArray, x)` or `pyconvert(Any, x)` will not use the rule. Hence we have well-scoped rules, avoid piracy, avoid cases where the conversion rules applied depend on which packages are loaded.

In particular, since passing Python objects to Julia in JuliaCall normally uses `pyconvert(Any, x)`, only rules created by the "owner" of `pytype(x)` are applied. This makes passing Python values around predictable - some third-party package defining their `list` to `MyArray` rule will not affect how `list` gets passed to Julia by default.

By ignoring the MRO of the passed Python object, we ignore issues with the arbitrary ordering of types in the MRO. Our proposal guarantees that if you have an applicable rule with `t=t1` then it can only be overridden by a later rule with `t=t2` if `t2` is a strict subclass of `t1`. Currently it can be overridden if `t2` is completely unrelated but just happens to be higher up the MRO.

I think this scheme is sufficiently general to encode rules in the priority order users will want. When adding a rule, you must own `t` or `S`. If you own `t` then it will be more specific than anything else anyway. If you own `S` then you have to opt in to using the rule like `pyconvert(S, x)` in which case only your rules pass the filter. Or if you do `pyconvert(Union{Foo,S}, x)` then whether you get a `Foo` or an `S` depends on insertion order, but if `Foo` came from a parent package, then you should rightly get a `Foo`, which will be the case because it's rules were defined first. So basically insertion order prevents overwriting rules from earlier-loaded packages. This does mean the output type can be import-order-dependent, but only where there are unions, and this case is inherently ambiguous so we have to pick something arbitrarily anyway.

## What about `jlwrap` and `array`?

We will have rules like `t=juliacall.AnyValue, T=S=Any` and `t=, T=PyArray, S=Any`. Provided we define these first, they will be applied first unless a rule for a more specific `t` is defined.

## Worked examples

Here are some rules for `t=list`:
- `T=PyArray, S=Any`: canonical conversion to a `PyArray`, used if you specify converting to `PyArray` or `AbstractArray` or `Any`.
- `T=Array, S=DenseArray`: used if you specify converting to `Array` or `DenseArray`, but `AbstractArray` gets you a `PyArray`.
- `T=Set, S=AbstractSet`: used if you specify converting to `Set` or `AbstractSet`.
- `T=Tuple, S=Tuple`: used if you specify converting to `Tuple`.

Some rules for `t=None`:
- `T=Nothing, S=Any`: canonical
- `T=Missing, S=Missing`: specify `Missing` (or `Union{Missing, Foo}`)

Some rules for `t=float`:
- `T=Float64, S=Any`: canonical
- `T=Float32, S=Float32`: specify another float type
- `T=Number, S=Number`: specify another non-float number type such as `Integer`
- `T=Missing, S=Missing` (for NaN)
- `T=Nothing, S=Nothing` (for NaN)

Some examples for converting a `float`:
- to `Any`: only rule 1 applies (filtering on `S`)
- to `Float32`: rules 2 and 3 apply (rule 1 ignored due to `T`, others due to `S`) so rule 2 is tried first.
- to `Integer`: only rule 3 applies (filtering on `S`).
- to `Union{Integer, Missing}`: rules 3 and 4 apply (filtering on `S`) so rule 3 is tried first.

Say some package defines `myfloat <: float` and adds a rule for it:
- `T=BigFloat, S=Any`

Examples converting a `myfloat`:
- to `Any`: rule 1 and the new rule apply. New rule more specific in `t`, so use new rule.
- to `AbstractFloat` or `Number`: pretty much the same.
- to `BigFloat`: only the new rule applies.
- to `Float32`: new rule doesn't apply, so as above rule 2 is first.

## Pros and cons

Pros:
- Strict ownership of rules - avoids piracy.
- Return type of `pyconvert` more predictable.
- Clearer semantics/rule ordering than currently.
- The number of applicable rules is massively cut down by filtering on `S` (usually to 1).
- Where there are more than 1, ordering by `t` should then be mostly unique. Using insertion order is mainly to disambiguate unions, plus special rules like for buffers and jlwrap.
- Easy to "opt in" to a conversion rule by being more specific about what you are converting to (see the `MyArray` example above).

Cons:
- People might still pirate (i.e. make rules with `S=Any` for which they don't own `t`).
- `pyconvert(Union{AbstractArray,MyArray}, x)` does not do what you might expect (use the generic `AbstractArray` rules plus the special `MyArray` rule) because the union gets normalised down to `AbstractArray` first, so the `MyArray` rule is never considered. You need to take more specific unions like `Union{PyArray,Array,MyArray}` which is annoying. We could make a helper function to create such a union for you.

## Rejected ideas

- I considered ordering also based on specificity of `T` (more specific wins) and `S` (less specific, i.e. more canonical, wins).

If we use `S` then in the `float` example we prefer the generic `Number` rule over the specific `Float32` rule. But if we use `T` then a `juliacall.DictValue` has rules `t=juliacall.AnyValue, T=Any, S=Any` and `t=Mapping, T=Dict, S=Any` and the latter rule will be preferred, which isn't what we want.

The current proposal only alters ordering a little - namely by removing priority and ignoring MRO - and relying on more aggressive filtering from `S`.

- We could allow `add_rule` to not always add at the end of the list. It could specify one or more existing rules that it must appear above. Rejected because as explained in the discussion section, the existing proposal is sufficient for any sensible rule definitions.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.