msgspec / msgspec/msgspec

Is msgspec suitable for high-performance serialization of binary data?

Open
#174 6 comments 8 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.1k
Forks
193
Avg merge
3d 20h
Merged PRs (30d)
15

Description

I have a small number of large numpy arrays (1-100MB total) stored in a structured way, with some additional JSON metadata. Something like:

{
    "my_array": numpy.array([0, 1, 2, ...]),
    "list_of_array": [numpy.array([0, 0, 0]), numpy.array([1, 1, 1]), ...],
    "dict_of_arrays": {"a": numpy.array([0, 0, 0]), "b": numpy.array([1, 1, 1]), ...},
    "some_metadata": "blah",
    "some_other_metadata": {"a": 1, "b": 2}
}

Let's say I want to send data structures like that over a socket (e.g., zeromq). I could pickle, but would prefer a serialization format that's more version- and language-agnostic.

Is msgspec (in JSON or msgpack mode) suitable for this use case? Are there any other considerations I should keep in mind? (I know a couple years ago a lot of work went into pickle5/dask comms to make serialization zero-copy and fast for this kind of use case.)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the issue's examples of numpy arrays, JSON or MessagePack encoding, and socket or ZeroMQ transfer. Determine whether the repository identifies an existing entry point or test for binary-array serialization; done would require a defined, version- and language-agnostic approach with its performance and compatibility considerations documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.