StackStorm / StackStorm/st2

Move from pickle to JSON serializer for message bus messages

Open
#1,761 11 comments 0 reactions 1 assignee View on GitHub

@Kami is already working on this.

Since Feb 8, 2018.

enhancement
Dominant language
Python
Stars
6.5k
Forks
787
PR merge metrics
No merged PRs in 30d

Description

(I've mentioned this multiple times before, but now I'm creating ticket so we can track / prioritize it).

We should move from pickle which we use right now to the JSON serializer for the messages which are dispatched on the RabbitMQ message bus.

Using pickle is fine if all data which goes through the message bus is controlled and generated by us (and if you don't mind about cross Python version and / or cross programming language compatibility), but this is not the case here (triggers can contain arbitrary data since we poll 3rd systems, etc.). This caries a big risk since if data is maliciously constructed.

Switching away from pickle means we won't be able to directly dispatch class instances to the messages bus anymore and we will need to use something like Thrift / Protobus and define structure for all the classes and data we want to pass through the message bus. An alternative would be to write a custom JSON serializer for all the classes we want to dispatch on the message bus, but this requires more work and is less portable then using something like Thrift.

That's actually a good thing since those classes will now be decoupled from Python and we will be able to use Thrift / Protobuf definitions to work with them in other languages (e.g. JavaScript).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.