canonical / canonical/data-platform-libs
[Model] Make (de)serialization a first class feature
- Dominant language
- Python
- Stars
- 13
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
Currently, we have two issues that look like intertwined: (1) databag format changes between charm revisions and we need to handle that at upgrades; and (2) predictable serialization, so we make sure a given class with the same data always get transformed into exactly the same string.
The later problem will happen every time we change the way we handle databags. In upgrades, we will also have to provide a way to upgrade from "model-revision-X" to "model-revision-X+1" and as @MiaAltieri highlighted, they will have to coexist while the upgrade is in-progress.
The former is an old issue that recurrently bubbles up in Juju. As we start to build more complex data structures that are serialized to the databag, we cannot guarantee anymore that the final string remains the same, even if the original struct had exactly the same data.
# Proposal
To have a `BaseModel` class that can cover both scenarios.
This class would look like:
```
class RootDataPlatformModel(BaseModel):
revision: int
@root_validator
def convert_data_following_rev(...):
# Sequence of if-elif-else that format the data according to the the incoming revision code
def __str__(...):
data = self.model_dump() # define custom serializers with @model_serializer
# Now, iterate over the revisions this class knows about: we must create a final string that follows the `revision` field
data_following_rev = ....
# Now that we know the data formatted as a dict for revision X, we should iterate over its structure and start converting
# each piece to string
for key, value in data_following_rev.items():
# For each literal type, the conversion to string is already done
# For each object within the "key" or "value" that implements RootDataPlatformModel, call str(this object)
# For structures like raw Lists, Sets, Tuples, we need to make sure they are ordered
```
## Using Metaclasses to Implement Different Revisions
This code will rapidly evolve into a complex web of `if-elif-else` as the number of exceptions between revisions grow. ideally, we do not want to deal with object "RootDataPlatformModel", but rather "RootDataPlatformModelRevisionX" when we are working on revision X.
So, what we could do instead is have a metaclass to decide which final object we will create, based on the revision:
```
class RootModelSelector(type):
def __new__(cls, args, kwargs):
# This method will return a different subclass of `BaseModel` depending on which revision code has been provided
class RootDataPlatformModel(metaclass=RevisionSelector):
revision: int
@root_validator
def deserializer(...):
# Now, this method is drastrically simplified, as we will use
def __str__(...):
# Also simplified, we always serialize for the appropriate revision
```
And the way we'd use it is:
```
class PeerRelationRev95Model(RootPeerRelationModel):
...
class PeerRelationRev99Model(RootPeerRelationModel):
...
class PeerRelationSelector(RootModelSelector):
# Implements the __new__ to ensure we are we select anything as follows:
# if revision <= 95: return PeerRelationRev95Model()
# elif revision 95 < or <=99: return PeerRelationRev99Model()
class RootPeerRelationModel(BaseModel, metaclass=PeerRelationSelector):
revision: int
@root_validator
def deserializer(...):
# Now, this method is drastrically simplified, as we will use
def __str__(...):
# Also simplified, we always serialize for the appropriate revision
```
Any "upper class" calls exclusively the `RootPeerRelationModel`.
# How to Select the Revision
The actual selection of the revision is done by the upper classes that create `BaseModel` objects. These are the classes that have access to the databag and can also query the main charm to know which revision they should take into consideration.
In the case we are in the middle of an upgrade, databag revisions X and X+1 may coexist for sometime. Therefore, a class that decides to load the databag should first check if all its peers have finished moving to X+1. If not, then it should continue to use (de)serialization for revision X.
Once the upgrade is finished, then the units can use revision X+1 (de)serialization and persist that new format in the databag instead.
Contributor guide
Assessment
This issue has not been assessed yet.