On a decoder for `JsonCustomEncoder`
- Dominant language
- Python
- Stars
- 5.3k
- Forks
- 2.2k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 75
Description
### Description
`JsonCustomEncoder` allows for non-standard objects to be encoded in JSON. However, many of the outputs are just translated to lists, which makes it impossible to infer the origin type when decoding. For example:
```python
if isinstance(obj, (np.number, np.ndarray)):
return obj.tolist()
elif isinstance(obj, complex):
return [obj.real, obj.imag]
```
A 2-element ndarray and a complex number are **indistinguishable!**
The JSON implementation in python allows for ``object_hook`` to decode dictionaries, the implication being that any non-standard type can be encoded as a dictionary and then transparently decoded by an `object_hook` method.
JSON builds recursively, so sub-dictionaries, e.g. for the dtype of an ndarray will be built before the enclosing ndarray, so ``object_hook`` can be used on nested structures.
My proposal is to either deprecate `JsonCustomEncoder ` or just add a new JSON (en&de)coder that utilizes this dictionary structure. I include a reference implementation below that is easily extensible and would allow for various sub-packages in astropy to register in en/decodings. In the example an important part of the dictionary is ``type="..."``, which is very similar to the YAML ``!`` identifier.
### Example implementation
```python
class JSONExtendedEncoder(json.JSONEncoder):
_registry = [] # list of tuples
def default(self, obj):
for cls, func in self._registry:
if isinstance(obj, cls):
code = func(obj)
break
else:
code = super().default(obj)
return code
@classmethod
def register_encoding(cls, type):
def register(func):
# inserting subclasses before parent classes, so encountered first
for i, (key, _) in enumerate(cls._registry):
if issubclass(type, key):
cls._registry.insert(i, (type, func))
break
else: # put at the end
cls._registry.append((type, func))
return func
return register
def _base_encode(obj):
qualname = obj.__class__.__module__ + "." + obj.__class__.__qualname__
code = {"type": qualname}
return code
@JSONExtendedEncoder.register_encoding(np.ndarray)
def _encode_ndarray(obj):
code = _base_encode(obj)
code.update(value=obj.tolist(), dtype=str(obj.dtype)) # TODO! encode dtype
return code
@JSONExtendedEncoder.register_encoding(u.Quantity)
def _encode_quantity(obj):
code = _encode_ndarray(obj.value)
code["unit"] = obj.unit.to_string()
return code
```
```python
class JSONExtendedDecoder(json.JSONDecoder):
_registry = []
def __init__(self, *, parse_float=None, parse_int=None, parse_constant=None, strict=True):
super().__init__(object_hook=self.object_hook, parse_float=parse_float,
parse_int=parse_int, parse_constant=parse_constant,
strict=strict)
@classmethod
def object_hook(cls, code):
try:
qualname = code.pop("type").split(".")
module = importlib.import_module(".".join(qualname[:-1]))
constructor = getattr(module, qualname[-1])
except ModuleNotFoundError as e:
raise # TODO!
for key, func in cls._registry:
if issubclass(constructor, key):
obj = func(constructor, code.pop("value"), code)
break
else:
obj = code
return obj
@classmethod
def register_decoding(cls, type):
def register(func):
# inserting subclasses before parent classes, so encountered first
for i, (key, _) in enumerate(cls._registry):
if issubclass(type, key):
cls._registry.insert(i, (type, func))
break
else: # put at the end
cls._registry.append((type, func))
return func
return register
@JSONExtendedDecoder.register_decoding(u.Quantity)
def _decode_quantity(constructor, value, code):
return constructor(value, **code)
@JSONExtendedDecoder.register_decoding(np.ndarray)
def _decode_ndarray(constructor, value, code):
return constructor(value, **code)
```
```python
val = np.array([3], dtype=float) * u.km
dumped = json.dumps(val, cls=JSONExtendedEncoder)
print(val, dumped)
out = json.loads(dumped, cls=JSONExtendedDecoder)
print(out, type(out))
```
[3.] km {"type": "astropy.units.quantity.Quantity", "value": [3.0], "dtype": "float64", "unit": "km"}
[3.] km
Contributor guide
Research direction
Begin with JsonCustomEncoder and Python's json.JSONEncoder/object_hook APIs, then compare the reference JSONExtendedEncoder and JSONExtendedDecoder design in the issue. Done requires settling whether to deprecate JsonCustomEncoder or add the proposed extensible encoder/decoder, including how Astropy types register their encodings and decodings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100