imazen / imazen/imageflow-server

RFC: Transmission of binary data with JSON

Open
#60 1 comment 0 reactions 0 assignees View on GitHub
imageflow_server requesting-feedback
Dominant language
C#
Stars
316
Forks
37
PR merge metrics
No merged PRs in 30d

Description

There's already a good writeup on this topic: http://blog.marcinbudny.com/2014/02/sending-binary-data-along-with-rest-api.html

We need to send binary image data, often in excess of 10MB, between the client and server. JSON Base64 increases byte count by 33%, but also has decode/encode overhead.

Key questions for *your* platform/language:

1. How efficient is your JSON implementation and decoding and encoding enormous base64 strings? With Rust<->Rust we could may see 40ms overhead for 4 codings of a 10MB image.
2. Does your HTTP client support sending multipart requests and parsing multipart responses?

Please speak up if you lack both multipart support and a fast JSON codec.

## JSON implementations

... are free to [set limits on the maximum size of strings or the message itself](http://stackoverflow.com/a/30739035).

Vary widely in speed: https://www.reddit.com/r/programming/comments/3pojrz/the_fastest_json_parser_in_the_world/

Are ubiquitous and usually optimized. Benchmarks of huge string parsing and base64 decoding appreciated.

## BSON

BSON has a native binary datatype, but implementations tend to be less optimized than their JSON counterparts. Benchmarks of huge binary blobs appreciated.

## HTTP libraries

Benchmarks of multipart parsing overhead on your platform would be appreciated.

## When it doesn't matter so much

* We can specify URLs instead of sending input blobs, and offload that fetch (and potentially, have some kind of caching).
* We can ask the server to just send the first image result instead of using multipart. We still have to handle failure JSON, though.

## Non-HTTP interactions

* libimageflow users can send pointers to buffers.
* imageflow_tool can read/write to the local filesystem.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start with the linked writeup, then gather the requested benchmarks for JSON/Base64 and multipart handling on the repository's C# platform; done means documenting a selected transport approach and its trade-offs.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, json
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.