internetarchive / internetarchive/openlibrary

kTLS with Docker and Nginx

Open
#7,122 0 comments 5 reactions 0 assignees View on GitHub
Lead: @cdrini Priority: 3 Theme: Performance Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

I was wondering if it might be worth enabling [kTLS](https://www.kernel.org/doc/html/latest/networking/tls-offload.html) on any Nginx servers that serve a lot of static content or cached responses.

I was looking at how Netflix can [serve 800Gb/s of TLS encrypted video content from a single server](https://people.freebsd.org/~gallatin/talks/euro2022.pdf), and a large part of it appears to be kTLS.

I also saw a blog post from Nginx about improving [Nginx performance with Kernel TLS and SSL_sendfile()](https://www.nginx.com/blog/improving-nginx-performance-with-kernel-tls/) using recent versions of Linux and FreeBSD, so I gave it a shot in Linux and Docker.

I did a test similar to the (curiously broken) bash loop in the blog post, and with kTLS enabled my CPU usage went from around 48-50% to around 27-29% while serving a 1 GB image 100 times, and throughput about doubled.

As it was just a test I didn't put a huge amount of effort into getting things working properly, so although I tried to load the kernel module from within `docker-compose.yml` using `cap_admin`, I ended up just loading it on the host with `sudo modprobe tls`.

I've attached the files I used, including some a dummy SSL cert.

[nginx_ktls.zip](https://github.com/internetarchive/openlibrary/files/9935640/nginx_ktls.zip)

Just raising the idea for evaluation.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the attached nginx_ktls.zip and its docker-compose.yml, then read the linked Nginx and kernel TLS references. Check whether the proposed setup fits Open Library's deployment and static or cached response serving; done means a maintainer-approved evaluation or a clearly scoped implementation path.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux, nginx
Domain
devops, infrastructure, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.