Support different de/serialization strategies

Open
#478 4 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
haskell
Domain
distributed-systems

Research direction

Start by reading the current Process abstraction and the send entry point to understand how Binary-based serialization is wired. Compare that API with the proposed Serialize strategy and Servant's MimeRender/MimeUnrender; done would mean agreeing on the abstraction and updating the affected types and examples without breaking serialization-agnostic programs.

Written by the indexing model from the issue text.

Description

discuss enhancement Feature Request

Currently, all messages are de/serialized via the Binary instance.

This is an implementation detail; there is strict required beyond the existence of two functions, encode :: a -> ByteString and decode :: ByteString -> Either String a. However, discussions in the DataHaskell group have raised the point that serialization for data science structures (e.g. dataframes) could be made more efficient by leveraging Parquet over the wire.

I'm thinking of an extension of the current Process monad, to support also a de/serialization strategy. This is strongly inspired by Servant's MimeRender and MimeUnrender classes. Consider:

newtype Process strategy a = Process ... -- Monad is unchanged, type has a new `strategy`

class Serialize strategy a where
    encode :: Proxy strategy -> a -> ByteString
    decode :: Proxy strategy -> ByteString -> Either String a

An example for JSON would be:

import Data.Aeson

data JSON

class (ToJSON a, FromJSON a) => Serialize JSON a where
    encode _ = encode
    decode _ = eitherDecodeStrict

Then, for the actual message sending e.g. via send, the type would become:

send :: Serialize strategy a => ProcessId -> a -> Process strategy ()

Then, programs that are agnostic of serialization are of type Process strategy a (instead of Process a), while programs which require a specific serialization method, such as JSON, will be of type Process JSON a.

Dominant language
Haskell
Stars
751
Forks
99
Avg merge
45m
Merged PRs (30d)
2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from haskell-distributed/distributed-process

All issues in haskell-distributed/distributed-process

Similar issues

More Haskell issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.