Database Manual / Self-Managed Deployments / Administration / Performance

TCMalloc Performance Optimization for a Self-Managed Deployment

Changed in version 8.0.

Starting in MongoDB 8.0, MongoDB uses an upgraded version of TCMalloc that uses per-CPU caches, instead of per-thread caches, to reduce memory fragmentation and make your database more resilient to high-stress workloads.

To use the new TCMalloc version:

Platform Support

Operating systems that support MongoDB 8.0 also support the updated TCMalloc, except for the following:

  • RHEL 8 / Oracle 8 on the PPC64LE and s390x architectures

  • RHEL 9 / CentOS 9 / Oracle 9 on the PPC64LE architecture

These operating systems use the legacy TCMalloc version. If you use these operating systems, disable THP.

Windows uses the legacy TCMalloc version and does not support the updated TCMalloc.

Enable Transparent Hugepages (THP)

Transparent Hugepages (THP) is a Linux memory management system that reduces the overhead of Translation Lookaside Buffer (TLB) lookups. THP achieves this by combining small pages and making them appear as larger memory pages to the application.

In MongoDB 8.0 and later, ensure that THP is enabled before mongod starts by creating a service file for your platform's initialization system. If you are using MongoDB 7.0 or earlier, disable THP.

Note

THP operates at a system level, so it applies to all processes under the system. You cannot enable or disable THP per-process level.

Additionally, for RHEL and CentOS systems that use ktune and tuned performance profiles, you must also create a custom tuned profile.

Create a Service File

To create a service file that enables THP, use the built-in initialization system for your platform. Recent versions of Linux typically use systemd, which uses the systemctl command. Older versions of Linux tend to use System V init, which uses the service command. For more information, see the documentation for your operating system.

Use the initialization system for your platform:

1

Create the systemd unit file

Create the following file and save it at /etc/systemd/system/enable-transparent-huge-pages.service:

[Unit]
Description=Enable Transparent Hugepages (THP)
DefaultDependencies=no
After=sysinit.target local-fs.target
Before=mongod.service

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo always | tee /sys/kernel/mm/transparent_hugepage/enabled > /dev/null && echo defer+madvise | tee /sys/kernel/mm/transparent_hugepage/defrag > /dev/null && echo 0 | tee /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none > /dev/null && echo 1 | tee /proc/sys/vm/overcommit_memory > /dev/null'

[Install]
WantedBy=basic.target

Note

Some versions of Red Hat Enterprise Linux, and potentially other Red Hat-based derivatives, use a different path for the THP enabled file:

/sys/kernel/mm/redhat_transparent_hugepage/enabled

Verify which path is in use on your system and update the enable-transparent-huge-pages.service file accordingly.

2

Reload systemd unit files

To reload the systemd unit files and make enable-transparent-huge-pages.service available for use, run the following command :

sudo systemctl daemon-reload
3

Start the service

Run:

sudo systemctl start enable-transparent-huge-pages

To verify that the relevant THP settings have changed, run the following command:

cat /sys/kernel/mm/transparent_hugepage/enabled && cat /sys/kernel/mm/transparent_hugepage/defrag && cat /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none && cat /proc/sys/vm/overcommit_memory

On Red Hat Enterprise Linux and potentially other Red Hat-based derivatives, you may instead need to use the following:

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled && cat /sys/kernel/mm/redhat_transparent_hugepage/defrag && cat /sys/kernel/mm/redhat_transparent_hugepage/khugepaged/max_ptes_none && cat /proc/sys/vm/overcommit_memory

The output should resemble the following:

always
defer+madvise
0
1
4

Configure your operating system to run it on boot.

To ensure that this setting is applied each time the operating system starts, run the following command:

sudo systemctl enable enable-transparent-huge-pages
5

(Optional) Customize tuned or ktune profile

If you use tuned or ktune proffiles on RHEL/ CentOS, you must also create a custom tuned profile.

1

Create the init.d script

Create the following file and save it at /etc/init.d/enable-transparent-hugepages:

#!/bin/bash
### BEGIN INIT INFO
# Provides:          enable-transparent-hugepages
# Required-Start:    $local_fs
# Required-Stop:
# X-Start-Before:    mongod mongodb-mms-automation-agent
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Enable Linux Transparent Hugepages
# Description:       Enable Linux Transparent Hugepages, to improve
#                    database performance.
### END INIT INFO

case $1 in
  start)
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/transparent_hugepage
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/redhat_transparent_hugepage
    else
      return 0
    fi

    echo 'always' | tee ${thp_path}/enabled > /dev/null && echo defer+madvise | tee ${thp_path}/defrag > /dev/null && echo 0 | tee ${thp_path}/khugepaged/max_ptes_none > /dev/null && echo 1 | tee /proc/sys/vm/overcommit_memory > /dev/null'

    unset thp_path
    ;;
esac
2

Make the script executable

Run:

sudo chmod 755 /etc/init.d/enable-transparent-hugepages
3

Run the script

Run:

sudo /etc/init.d/enable-transparent-hugepages start

To verify that the relevant THP settings have changed, run the following command:

cat /sys/kernel/mm/transparent_hugepage/enabled && cat /sys/kernel/mm/transparent_hugepage/defrag && cat /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none && cat /proc/sys/vm/overcommit_memory

On Red Hat Enterprise Linux and potentially other Red Hat-based derivatives, you may instead need to use the following:

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled

The output should resemble the following:

always
defer+madvise
0
1
4

Configure your operating system to run it on boot

To ensure that this setting is applied each time the operating sytem starts, run the following command for your Linux distribution:

DistributionCommand

Ubuntu and Debian

sudo update-rc.d enable-transparent-hugepages defaults

SUSE

sudo insserv /etc/init.d/enable-transparent-hugepages

Red Hat, CentOS, Amazon Linux, and derivatives

sudo chkconfig --add enable-transparent-hugepages
5

(Optional) Customize tuned or ktune profile

If you are using tuned or ktune profiles on RHEL/ CentOS, you must also create a custom tuned profile.

Using tuned and ktune

Important

If you use tuned or ktune, perform the steps in this section after creating the service file.

tuned and ktune are kernel tuning utilities that can affect the Transparent Hugepages setting on your system. If you are using tuned or ktune on your RHEL or CentOS system while running mongod, you must create a custom tuned profile to ensure that THP stays enabled.

1

Create a new profile

Create a new directory to store the custom tuned profile. The following example inherits from the existing virtual-guest profile, and uses virtual-guest-thp as the new profile:

sudo mkdir /etc/tuned/virtual-guest-thp
2

Edit tuned.conf

Create and edit /etc/tuned/virtual-guest-thp/tuned.conf. Add the following text:

[main]
include=virtual-guest

[vm]
transparent_hugepages=always

This example inherits from the existing virtual-guest profile. Select the appropriate profile for your system.

3

Enable the new profile

Run:

sudo tuned-adm profile virtual-guest-thp

Enable Per-CPU Caches

To verify that TCMalloc is running with per-CPU caches, ensure that:

If per-CPU caches aren't enabled, ensure that:

  • You disable glibc rseq.
  • You're using Linux kernel version 4.18 or later.

Disable glibc rseq

The new TCMalloc requires Restartable Sequences (rseq) to implement per-CPU caches. If another application, such as the glibc library, registers an rseq structure before TCMalloc, TCMalloc can't use rseq. Without rseq, TCMalloc uses per-thread caches, which are used by the legacy TCMalloc version.

To ensure that TCMalloc can use rseq to enable per-CPU caches, you can disable glibc’s registration of a rseq structure. To disable glibc rseq, set the following environment variable before you start mongod:

GLIBC_TUNABLES=glibc.pthread.rseq=0
export GLIBC_TUNABLES

Check Kernel Version

If you disabled glibc rseq and per-CPU caches are still not enabled, ensure that you're using Linux kernel version 4.18 or later. To check your kernel version, run the following command:

uname -r

Learn More