Update reverse proxy recommendations for Argus and C... | Joystream/joystream

As established in https://github.com/Joystream/joystream/issues/3808, we should update the documentation of Argus and Colossus nodes to include recommended reverse-proxy setup that supports HTTP/2 connections.

Note that there is already a section that includes an example Nginx configuration in the Argus docs: https://github.com/Joystream/joystream/blob/master/distributor-node/docs/node/index.md#example-nginx-configuration, but it's for HTTP 1.1

Scope of work:

Research possible reverse-proxy settings for at least Nginx and Caddy (+ possibly other reverse-proxy servers) related to HTTP/2
Run some benchmarks to establish the benefits of switching from HTTP 1.1 to HTTP/2 for reverse-proxy connections for both Colossus and Argus, using different reverse-proxy servers and configurations
Come up with a set of optimal, recommended reverse-proxy settings for the nodes, including request/connection limits that would mitigate potential DoS attacks
Update the documentation of both Argus and Colossus w/ the recommended reverse-proxy configuration for HTTP/2

Benchmarking:

Setup

For the purpose of benchmarking the minimum recommended setup is:

Machine A: running joystream node + query node + storage node
Machine B: running joystream node + query node + distributor node
Machine C: running the benchmarks

Provided that:

There is non-zero latency between A-B and B-C (the machines need to be geographically spread out)
Joystream nodes on machine A and B form a single blockchain
Query node on machines A and B connects to a local joystream node and distributor/storage node connect to a local query node

Storage and distribution buckets needs to be set up. In order to do this, following commands can be executed (either on Machine A or Machine B):

export COLOSSUS_1_URL="https://your-storage-node-public-url.com"
export DISTRIBUTOR_1_URL="https://your-distributor-node-public-url.com"

yarn workspace network-tests run-test-scenario initStorageAndDistribution

Now, provided you're already running local joystream and query-node via docker-compose on both machine A and machine B, you can start storage/distributor nodes via:

Machine A:

docker-compose up -d colossus-1

Machine B:

docker-compose up -d distributor-1

In order to do the benchmarking you'll need some mock data objects. To generate them, execute the following command on either machine A or B:

cd distributor-node
./bin/run dev:batchUpload -B 0 -C 10 -S 100 -b static:council -e https://your-storage-node-public-url.com -y

Browser benchmarking

In order to do some benchmarking within a browser, a simple html page can be created:

<html>
    <body>
        <div></div>
        <script>
            let content = ''
            for(i=0; i<100; ++i) {
                content += `<img src="https://your-distributor-node-public-url.com/api/v1/assets/${i}" />`
            }
            document.getElementsByTagName('div')[0].innerHTML = content;
            let start = Date.now();
            window.onload = () => alert(`Loading took: ${Date.now() - start}`);
        </script>
    </body>
</html>

This is just an example containing a script that insert 100 <img> elements into the page (forcing the browser to request 100 different assets from the distributor node) and measure the time it took to load the page.

Remember that the browser cache needs to be disabled or cleared each time after loading the page to prevent client-side caching from affecting the results.

Remember that distributor node also has it's own cache, so on the first page load it will fetch all the assets from the storage node, but on the second page load it will serve the assets from it's own cache. Both ways of serving the assets should be benchmarked (in order to test non-cached cases, either the Argus cache needs to be cleaned, or different assets requested each time).

I recommend to run multiple tests of each scenario and calculate the average/median result, for example:

5x fetch non-cached assets from the distributor node with Client --http/2--> Argus --http/2--> Colossus
5x fetch non-cached assets from the distributor node with Client --http/2--> Argus --http/1.1--> Colossus
5x fetch non-cached assets from the distributor node with Client --http/1.1--> Argus --http/2--> Colossus
5x fetch non-cached assets from the distributor node with Client --http/1.1--> Argus --http/1.1--> Colossus
5x fetch cached assets from the distributor-node with Client --http/1.1--> Argus
5x fetch cached assets from the distributor-node with Client --http/2 --> Argus

Calculate the averages/medians and compare the results.

Command line benchmarking

Tools like h2load can be used to perform benchmarking outside of the browser context:

sudo apt-get install nghttp2-client
URLS=`
for ASSET in {0..1000}
do
	echo "https://your-distributor-node-public-url.com/api/v1/assets/${ASSET}"
done
`
h2load -n10000 -c10 -m20 ${URLS}       # HTTP/2 used by default
h2load -n10000 -c10 -m20 --h1 ${URLS}  # --h1 forces HTTP/1.1

In this example we're requesting the same 1000 assets (0-999) from distributor node over 10 separate connections, using --max-concurrent-streams=20 (for HTTP/1.1 max concurrent streams specifies the number of pipelining requests in-flight), first using http/2 and then http/1.1.

Update reverse proxy recommendations for Argus and Colossus to support HTTP/2

Description

Scope of work:

Benchmarking:

Setup

Browser benchmarking

Command line benchmarking

Contributor guide

First steps

Assessment

Update reverse proxy recommendations for Argus and Colossus to support HTTP/2

Description

Scope of work:

Benchmarking:

Setup

Browser benchmarking

Command line benchmarking

Contributor guide

First steps

Assessment

Get new issues in your inbox