Change default page table size to 4

Open
#4,375 20 comments 27 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
c, linux

Research direction

Start by inspecting the arm64 kernel configuration and the CONFIG_PGTABLE_LEVELS entry, then reproduce the failure with docker run --rm envoyproxy/envoy:v1.17.0 --version on the reported Raspberry Pi setup. Determine whether changing the default to 4 preserves supported page configurations; done means the kernel builds and Envoy starts without the tcmalloc allocation error.

Written by the indexing model from the issue text.

Description

Describe the bug
Google's tcmalloc assumes a page table size of 4 (which seems to be considered the default) to allocate heap and fails since the kernel is compiled with CONFIG_PGTABLE_LEVELS=3 by default.

To reproduce
As example you could try to run the envoy proxy which uses tcmalloc under the hood.
docker run --rm envoyproxy/envoy:v1.17.0 --version
This will fail with a memory allocation error.

Expected behaviour
The heap allocation should not fail and enovy should print its version information. This can be achieved by setting CONFIG_PGTABLE_LEVELS=4.

Actual behaviour
The heap allocation fails leading to envoy not being able start up and fail with an error message.

System

Raspberry Pi 4 Model B Rev 1.4
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"

Raspberry Pi reference 2020-08-20
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 7252c154838ec5b4576f29c996ac8fe3750cae12, stage2

Linux raspberrypi-4 5.10.17-v8+ #1421 SMP PREEMPT Thu May 27 14:01:37 BST 2021 aarch64 GNU/Linux
Revision	: d03114
Model		: Raspberry Pi 4 Model B Rev 1.4
Throttled flag  : throttled=0x0
Camera          : supported=0 detected=0

version 7d9a298cda813f747b51fe17e1e417e7bf5ca94d (clean) (release) (start)

Logs
This is the error message you will receive from envoy / tcmalloc when kernel is compiled with CONFIG_PGTABLE_LEVELS=3:

external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:550] MmapAligned() failed (size, alignment) 1073741824 1073741824 @ 0x55867b9470 0x55867aba14 0x55867ab454 0x5586794e94 0x55867a8684
external/com_github_google_tcmalloc/tcmalloc/arena.cc:34] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size) 131072 48 @ 0x55867b9780 0x5586794f10 0x55867a8684

Additional context
This is a conversation from the OpenJDK project which had to deal with the same situation: http://openjdk.5641.n7.nabble.com/ZGC-aarch64-Unable-to-allocate-heap-for-certain-Linux-kernel-configurations-td420728.html
Basically their solution is to try allocating memory multiple times until it succeeds, which does not seem to be ideal.

This is the excerpt from the kernel documentation:

AArch64 Linux uses either 3 levels or 4 levels of translation tables
with the 4KB page configuration, allowing 39-bit (512GB) or 48-bit
(256TB) virtual addresses, respectively, for both user and kernel. With
64KB pages, only 2 levels of translation tables, allowing 42-bit (4TB)
virtual address, are used but the memory layout is the same.

These are the open issues for envoy and tcmalloc

Dominant language
C
Stars
13.2k
Forks
5.5k
Avg merge
2d 21h
Merged PRs (30d)
21

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from raspberrypi/linux

All issues in raspberrypi/linux

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.