Remove support for JSON serialization format
Dieses Issue hat noch niemand übernommen.
- Vorherrschende Sprache
- Python
- Sterne
- 20.6k
- Forks
- 3.3k
- PR-Merge-Kennzahlen
- PR-Kennzahlen ausstehend
Beschreibung
Now that we have a new binary fixed format for serialization, it would be good to eventually drop the old JSON based serialization code, since maintaining duplicate implementations slows down development, and there is a risk that the implementations will get out of sync.
At work we have some tools that consume the JSON based cache format, and it would be non-trivial to update them to directly use the binary format. As a workaround, we could provide a tool that converts a binary cache into JSON files that resemble the current JSON format (they don't need to be 100% compatible since the format isn't documented anywhere).
To implement this, we can make the binary format "self-describing", i.e. it would have enough redundancy that we can write a simple generic parser that reads arbitrary data serialized using the format and converts it into JSON, and that doesn't need to know too many details of each possible object type that can be serialized (to simplify maintenance).
Here is one possible to way to make the format self-describing:
- Each value/object is serialized as
<type tag><data>, where the format of data depends on the type tag. The type tag is a 8-bit integer. - Simple values are also encoded as a type tag followed by data. These include integers, strings, booleans, floats and
None. - We can have similar generic encoding for lists and dicts, such as
<type tag for list><number of items><arbitrary value>.... - AST nodes would also have distinct type tags, so that it would be possible to mix simple values and AST nodes.
- I would propose that AST nodes and types would be encoded using a format like this:
<type tag><field tag><arbitrary value>...<end tag>. Field tags are also 8-bit integers that map to JSON keys. The end tag is a reserved 8-bit number that is distinct from all type tags.
- I would propose that AST nodes and types would be encoded using a format like this:
Now we can implement a generic parser. It needs to know how to parse all simple values and container values, and it must have a mapping from valid type tags for AST nodes (including type objects) and the field tags to strings.
I think that this format would also make it easy to implement lazy deserialization. We can easily find the end of an arbitrary object in a serialized byte stream, without having to deserialize it. Instead of deserializing a FuncDef, for example, we could just find the and of the serialized representation and put the serialized byte string into a symbol table.
cc @ilevkivskyi
Beitragsleitfaden
Erste Schritte
- Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
- Forke das Repository und arbeite in einem Branch.
- Öffne einen Pull Request, der die Issue-Nummer nennt.
Rechercherichtung
Das Issue nennt keine Dateien, Tests oder Einstiegspunkte; beginne damit, die Implementierungen der JSON-Serialisierung und der binären Serialisierung von mypy zu finden. Kläre den Umfang der Migration und der Kompatibilität mit den Maintainer:innen, bevor du ein selbsbeschreibendes Format oder ein Konvertierungstool entwirfst, da das Issue Alternativen statt einer klar abgegrenzten Aufgabe präsentiert.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- python
- Bereich
- tooling
- Issue-Typ
- Refactoring
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Aktivitätsstatus
- Veraltet
- Klarheit
- Muss geklärt werden
- Anfängerfreundlichkeit
- 20/100