apache / apache/couchdb

Add support for MessagePack, or other compact/binary format

Open
#2,203 12 comments 0 reactions 0 assignees View on GitHub
enhancement patches-welcome
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

JSON is great for interoperability, but not so much for efficient network communication.

I would love to see a future version of CouchDB that supports a more compact data streaming format, such as MessagePack. (Ideally in both CouchDB and PouchDB, as PouchDB is really where it would be the most useful--but it needs CouchDB support, first).

I haven't looked deeply into it, but I expect MessagePack would be a good choice, as can fairly cleanly be converted 1:1 to/from JSON.

## Desired Behaviour

This should not, in any way, replace JSON as the primary content type used by CouchDB. I would see it becoming an optional `Accept:` and `Content-Type:` header. The conversion should be trivially handled in a middleware, so it shouldn't need to affect any core logic, and would make for an easy prototype implementation entirely outside of CouchDB.

So for incoming requests (POSTs/PUTs, etc):

If the Content-Type header matches MessagePack, simply convert to JSON, then pass along to normal processing.

If the Accept: header matches MessagePack, perform the normal operation, then before streaming the response, convert from JSON to MessagePack.

## Additional context

- There may be some corner cases around data conversion that need addressing to ensure 100% compatibility, particularly around [large numbers](https://github.com/tinylib/msgp/wiki/The-MessagePack-Type-System#base-types).

- There's [some confusion](https://github.com/msgpack/msgpack/issues/194) about which MIME type to use for MessagePack. If this proposal and format are ultimately selected, a small amount of bike-shedding will need to be done.

- There may be other data formats to consider, as well. I'd love to hear other suggestions. (Although, IMO, it should be an existing, open standard, rather than inventing our own.)

- BSON might be a candidate, but is not as attractive, at first blush, as it's both larger than MessagePack, and it's not as easily converted 1:1 to/from JSON.

- As a first step, writing a stand-alone CouchDB proxy and a PouchDB plugin would probably be worth-while, to allow testing the functionality for compatibility and performance. If there is broader interest, I may try to work on this in time.

Contributor guide

Open the contributing guide

Research direction

The proposal concerns optional Accept and Content-Type handling for POSTs, PUTs, and streamed responses, with CouchDB support first and a possible PouchDB plugin or proxy. The format choice, MIME type, conversion edge cases, implementation location, and completion criteria remain undecided.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang, javascript
Domain
api, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.