astropy / astropy/astropy

On a decoder for `JsonCustomEncoder`

Open
#12,793 3 comments 0 reactions 0 assignees View on GitHub
Feature Request utils
Dominant language
Python
Stars
5.3k
Forks
2.2k
Avg merge
1d 19h
Merged PRs (30d)
75

Description

### Description

`JsonCustomEncoder` allows for non-standard objects to be encoded in JSON. However, many of the outputs are just translated to lists, which makes it impossible to infer the origin type when decoding. For example:

```python
if isinstance(obj, (np.number, np.ndarray)):
return obj.tolist()
elif isinstance(obj, complex):
return [obj.real, obj.imag]
```

A 2-element ndarray and a complex number are **indistinguishable!**

The JSON implementation in python allows for ``object_hook`` to decode dictionaries, the implication being that any non-standard type can be encoded as a dictionary and then transparently decoded by an `object_hook` method.
JSON builds recursively, so sub-dictionaries, e.g. for the dtype of an ndarray will be built before the enclosing ndarray, so ``object_hook`` can be used on nested structures.

My proposal is to either deprecate `JsonCustomEncoder ` or just add a new JSON (en&de)coder that utilizes this dictionary structure. I include a reference implementation below that is easily extensible and would allow for various sub-packages in astropy to register in en/decodings. In the example an important part of the dictionary is ``type="..."``, which is very similar to the YAML ``!`` identifier.

### Example implementation

```python
class JSONExtendedEncoder(json.JSONEncoder):
_registry = [] # list of tuples

def default(self, obj):
for cls, func in self._registry:
if isinstance(obj, cls):
code = func(obj)
break
else:
code = super().default(obj)

return code

@classmethod
def register_encoding(cls, type):
def register(func):
# inserting subclasses before parent classes, so encountered first
for i, (key, _) in enumerate(cls._registry):
if issubclass(type, key):
cls._registry.insert(i, (type, func))
break
else: # put at the end
cls._registry.append((type, func))
return func

return register

def _base_encode(obj):
qualname = obj.__class__.__module__ + "." + obj.__class__.__qualname__
code = {"type": qualname}
return code

@JSONExtendedEncoder.register_encoding(np.ndarray)
def _encode_ndarray(obj):
code = _base_encode(obj)
code.update(value=obj.tolist(), dtype=str(obj.dtype)) # TODO! encode dtype
return code

@JSONExtendedEncoder.register_encoding(u.Quantity)
def _encode_quantity(obj):
code = _encode_ndarray(obj.value)
code["unit"] = obj.unit.to_string()
return code
```

```python
class JSONExtendedDecoder(json.JSONDecoder):
_registry = []

def __init__(self, *, parse_float=None, parse_int=None, parse_constant=None, strict=True):
super().__init__(object_hook=self.object_hook, parse_float=parse_float,
parse_int=parse_int, parse_constant=parse_constant,
strict=strict)

@classmethod
def object_hook(cls, code):
try:
qualname = code.pop("type").split(".")
module = importlib.import_module(".".join(qualname[:-1]))
constructor = getattr(module, qualname[-1])
except ModuleNotFoundError as e:
raise # TODO!

for key, func in cls._registry:
if issubclass(constructor, key):
obj = func(constructor, code.pop("value"), code)
break
else:
obj = code

return obj

@classmethod
def register_decoding(cls, type):
def register(func):
# inserting subclasses before parent classes, so encountered first
for i, (key, _) in enumerate(cls._registry):
if issubclass(type, key):
cls._registry.insert(i, (type, func))
break
else: # put at the end
cls._registry.append((type, func))
return func

return register

@JSONExtendedDecoder.register_decoding(u.Quantity)
def _decode_quantity(constructor, value, code):
return constructor(value, **code)

@JSONExtendedDecoder.register_decoding(np.ndarray)
def _decode_ndarray(constructor, value, code):
return constructor(value, **code)
```

```python
val = np.array([3], dtype=float) * u.km
dumped = json.dumps(val, cls=JSONExtendedEncoder)
print(val, dumped)

out = json.loads(dumped, cls=JSONExtendedDecoder)
print(out, type(out))
```
[3.] km {"type": "astropy.units.quantity.Quantity", "value": [3.0], "dtype": "float64", "unit": "km"}
[3.] km

Contributor guide

Open the contributing guide

Research direction

Begin with JsonCustomEncoder and Python's json.JSONEncoder/object_hook APIs, then compare the reference JSONExtendedEncoder and JSONExtendedDecoder design in the issue. Done requires settling whether to deprecate JsonCustomEncoder or add the proposed extensible encoder/decoder, including how Astropy types register their encodings and decodings.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.