canonical / canonical/data-platform-libs

[Model] Make (de)serialization a first class feature

Open
#181 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
13
Forks
12
PR merge metrics
No merged PRs in 30d

Description

Currently, we have two issues that look like intertwined: (1) databag format changes between charm revisions and we need to handle that at upgrades; and (2) predictable serialization, so we make sure a given class with the same data always get transformed into exactly the same string.

The later problem will happen every time we change the way we handle databags. In upgrades, we will also have to provide a way to upgrade from "model-revision-X" to "model-revision-X+1" and as @MiaAltieri highlighted, they will have to coexist while the upgrade is in-progress.

The former is an old issue that recurrently bubbles up in Juju. As we start to build more complex data structures that are serialized to the databag, we cannot guarantee anymore that the final string remains the same, even if the original struct had exactly the same data.

# Proposal

To have a `BaseModel` class that can cover both scenarios.

This class would look like:
```
class RootDataPlatformModel(BaseModel):

revision: int

@root_validator
def convert_data_following_rev(...):
# Sequence of if-elif-else that format the data according to the the incoming revision code

def __str__(...):
data = self.model_dump() # define custom serializers with @model_serializer

# Now, iterate over the revisions this class knows about: we must create a final string that follows the `revision` field
data_following_rev = ....

# Now that we know the data formatted as a dict for revision X, we should iterate over its structure and start converting
# each piece to string
for key, value in data_following_rev.items():
# For each literal type, the conversion to string is already done
# For each object within the "key" or "value" that implements RootDataPlatformModel, call str(this object)
# For structures like raw Lists, Sets, Tuples, we need to make sure they are ordered

```

## Using Metaclasses to Implement Different Revisions

This code will rapidly evolve into a complex web of `if-elif-else` as the number of exceptions between revisions grow. ideally, we do not want to deal with object "RootDataPlatformModel", but rather "RootDataPlatformModelRevisionX" when we are working on revision X.

So, what we could do instead is have a metaclass to decide which final object we will create, based on the revision:

```
class RootModelSelector(type):
def __new__(cls, args, kwargs):
# This method will return a different subclass of `BaseModel` depending on which revision code has been provided

class RootDataPlatformModel(metaclass=RevisionSelector):

revision: int

@root_validator
def deserializer(...):
# Now, this method is drastrically simplified, as we will use

def __str__(...):
# Also simplified, we always serialize for the appropriate revision
```

And the way we'd use it is:
```
class PeerRelationRev95Model(RootPeerRelationModel):
...

class PeerRelationRev99Model(RootPeerRelationModel):
...

class PeerRelationSelector(RootModelSelector):
# Implements the __new__ to ensure we are we select anything as follows:
# if revision <= 95: return PeerRelationRev95Model()
# elif revision 95 < or <=99: return PeerRelationRev99Model()

class RootPeerRelationModel(BaseModel, metaclass=PeerRelationSelector):

revision: int

@root_validator
def deserializer(...):
# Now, this method is drastrically simplified, as we will use

def __str__(...):
# Also simplified, we always serialize for the appropriate revision
```

Any "upper class" calls exclusively the `RootPeerRelationModel`.

# How to Select the Revision

The actual selection of the revision is done by the upper classes that create `BaseModel` objects. These are the classes that have access to the databag and can also query the main charm to know which revision they should take into consideration.

In the case we are in the middle of an upgrade, databag revisions X and X+1 may coexist for sometime. Therefore, a class that decides to load the databag should first check if all its peers have finished moving to X+1. If not, then it should continue to use (de)serialization for revision X.

Once the upgrade is finished, then the units can use revision X+1 (de)serialization and persist that new format in the databag instead.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.