How to Enable Tiered Memory Protection with Memory QoS in Kubernetes v1.36

Introduction

Kubernetes v1.36 introduces a smarter way to manage container memory with the updated Memory QoS feature. This guide walks you through enabling and configuring tiered memory protection based on Pod QoS classes. You'll learn how to use the new memoryReservationPolicy field to control whether memory is hard-reserved (memory.min) or soft-protected (memory.low), giving you better control over node memory pressure and reducing OOM risks. Whether you're a cluster administrator or a developer, this step-by-step approach will help you optimize memory allocation for your workloads.

How to Enable Tiered Memory Protection with Memory QoS in Kubernetes v1.36

What You Need

Kubernetes v1.36 (or later) cluster with cgroup v2 enabled on all nodes.
Feature gate MemoryQoS must be enabled (alpha in v1.36).
kubelet configuration access (e.g., via KubeletConfiguration file or flags).
Understanding of Pod QoS classes: Guaranteed, Burstable, BestEffort.
Optional: Metrics endpoint access (/metrics) to observe memory reservation.

Step-by-Step Guide

Step 1: Enable the Memory QoS Feature Gate

The feature is alpha and requires explicit activation. Edit the kubelet configuration on each node (or use a centralized KubeletConfiguration resource):

Set featureGates.MemoryQoS: true in the kubelet config file (e.g., /var/lib/kubelet/config.yaml).
Restart the kubelet service: systemctl restart kubelet.
Verify the gate is active: check kubelet logs for MemoryQoS feature gate enabled or inspect /sys/fs/cgroup/kubepods.slice/ – if memory.high is set on cgroups, throttling is active.

Step 2: Configure memoryReservationPolicy

By default, enabling the gate only turns on memory.high throttling (a soft limit). To add memory protection, set the memoryReservationPolicy field in kubelet configuration:

None (default): No memory reservation (memory.min or memory.low). Only throttling via memory.high applies.
TieredReservation: The kubelet writes per-Pod cgroup values for memory.min or memory.low based on QoS class.

To enable tiered protection, add to the kubelet config:

memoryReservationPolicy: TieredReservation

Restart kubelet for changes to take effect.

Step 3: Understand Tiered Protection by QoS Class

Once TieredReservation is active, the kubelet assigns cgroup memory settings as follows:

Pod QoS Class	Reservation Type	Example (request 512 MiB)
Guaranteed	`memory.min` (hard)	`536870912` – kernel never reclaims; if contention, OOM kills others.
Burstable	`memory.low` (soft)	`536870912` – kernel protects under normal pressure, reclaims under extreme pressure to avoid system-wide OOM.
BestEffort	None	No reservation; memory fully reclaimable at any time.

This is a key improvement over v1.27, where every container with a memory request got memory.min, potentially locking too much memory and causing OOMs. Now only Guaranteed Pods get hard protection.

Step 4: Monitor Memory Reservation

Kubernetes v1.36 adds two alpha metrics on the kubelet’s /metrics endpoint. Enable them by ensuring the MemoryQoS feature gate is active:

kubelet_memory_qos_node_memory_min_bytes: Total memory.min across all Pods on the node.
kubelet_memory_qos_cgroup_memory_low_bytes: Total memory.low across all Pods (available in v1.36+).

Use curl or Prometheus to scrape: curl http://localhost:10249/metrics | grep memory_qos. These help you verify that only Guaranteed Pods use hard reservation and that Burstable Pods use soft protection.

Step 5: Adjust memoryThrottlingFactor (Optional)

The memory.high value is set based on memoryThrottlingFactor (default 0.9). This means the kubelet sets memory.high = memory request * 0.9. You can change this value in kubelet configuration to a float between 0 and 1. Lower values cause earlier throttling. Example:

memoryThrottlingFactor: 0.8

This is independent of reservation policy – throttling works regardless of memoryReservationPolicy.

Step 6: Handle Kernel Version Warning

Some older kernels have issues with memory.high. Kubelet logs a warning if the kernel version is below 5.4 (or certain patched versions). To avoid instability, ensure all nodes run a kernel that properly supports cgroup v2 memory accounting. Check with uname -r.

Tips and Best Practices

Start with throttling only: Before enabling TieredReservation, observe workload behavior under memory.high throttling. This helps you tune memoryThrottlingFactor and confirm workloads tolerate cgroup-level pressure.
Reserve headroom: If node memory is tight, avoid enabling TieredReservation until you have enough cushion. For example, with Burstable Pods requesting 7 GiB on an 8 GiB node, soft protection (memory.low) still preserves memory under normal load but allows reclamation under pressure – a safer default than hard reservation.
Test with Guaranteed Pods: Only add hard protection (memory.min) to critical, predictable workloads. Over-reserving with memory.min can starve other processes and trigger OOM kills.
Monitor OOM events: Use kubectl top pods and node-level tools like dmesg to watch for out-of-memory kills. Tiered protection reduces risk but does not eliminate it.
Upgrade carefully: If migrating from v1.27 where all requests became memory.min, tiered reservation may free up memory for BestEffort or system daemons. Verify node headroom after upgrade.
Use metrics for validation: Regularly check the memory reservation metrics to confirm you are not accidentally oversubscribing hard reservations.

By following these steps, you can leverage Kubernetes v1.36’s updated Memory QoS to protect workloads proportionally, reduce system-wide OOM risks, and make more efficient use of your cluster’s memory.

Tags: