Add support for MessagePack, or other compact/binary format
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
JSON is great for interoperability, but not so much for efficient network communication.
I would love to see a future version of CouchDB that supports a more compact data streaming format, such as MessagePack. (Ideally in both CouchDB and PouchDB, as PouchDB is really where it would be the most useful--but it needs CouchDB support, first).
I haven't looked deeply into it, but I expect MessagePack would be a good choice, as can fairly cleanly be converted 1:1 to/from JSON.
## Desired Behaviour
This should not, in any way, replace JSON as the primary content type used by CouchDB. I would see it becoming an optional `Accept:` and `Content-Type:` header. The conversion should be trivially handled in a middleware, so it shouldn't need to affect any core logic, and would make for an easy prototype implementation entirely outside of CouchDB.
So for incoming requests (POSTs/PUTs, etc):
If the Content-Type header matches MessagePack, simply convert to JSON, then pass along to normal processing.
If the Accept: header matches MessagePack, perform the normal operation, then before streaming the response, convert from JSON to MessagePack.
## Additional context
- There may be some corner cases around data conversion that need addressing to ensure 100% compatibility, particularly around [large numbers](https://github.com/tinylib/msgp/wiki/The-MessagePack-Type-System#base-types).
- There's [some confusion](https://github.com/msgpack/msgpack/issues/194) about which MIME type to use for MessagePack. If this proposal and format are ultimately selected, a small amount of bike-shedding will need to be done.
- There may be other data formats to consider, as well. I'd love to hear other suggestions. (Although, IMO, it should be an existing, open standard, rather than inventing our own.)
- BSON might be a candidate, but is not as attractive, at first blush, as it's both larger than MessagePack, and it's not as easily converted 1:1 to/from JSON.
- As a first step, writing a stand-alone CouchDB proxy and a PouchDB plugin would probably be worth-while, to allow testing the functionality for compatibility and performance. If there is broader interest, I may try to work on this in time.
Contributor guide
Research direction
The proposal concerns optional Accept and Content-Type handling for POSTs, PUTs, and streamed responses, with CouchDB support first and a possible PouchDB plugin or proxy. The format choice, MIME type, conversion edge cases, implementation location, and completion criteria remain undecided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- erlang, javascript
- Domain
- api, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100