Optimizing Pod Density on AWS EKS: How Many Pods Fit on a Node?

When deploying applications in Amazon Elastic Kubernetes Service (EKS), understanding how many pods can run on a single node is crucial for optimizing resource utilization and cost efficiency. The number of pods that can fit on an EKS node depends on various factors, including the EC2 instance type, network interface limits, and Kubernetes configuration.

Key Factors Influencing Pod Limits on EKS Nodes

1. Elastic Network Interfaces (ENIs) and IP Address Limits
   Each EC2 instance type supports a specific number of ENIs and IP addresses per ENI. In EKS, each pod is assigned a unique IP address, meaning the pod limit for a node is determined by the total number of available IPs. This is the primary restriction for pod density on EKS.

For example:

A t3.medium instance supports 3 ENIs, with 6 IPs per ENI, resulting in a maximum of 17 pods (16 for workloads and 1 reserved for the node itself).

A m5.large instance supports 3 ENIs, with 10 IPs per ENI, allowing up to 29 pods.

2. Instance Resources (vCPU and Memory)
While the ENI/IP limit sets the theoretical maximum, practical limits are often constrained by the instance’s vCPU and memory. Resource-intensive workloads may exhaust the node’s capacity before reaching the ENI-based pod limit. Kubernetes allocates CPU and memory for each pod based on its resource requests and limits, and the node itself reserves resources for system processes and the kubelet.

1. Kubelet Configuration: max-pods
   AWS provides a predefined max-pods limit for each instance type in EKS. This value is based on the ENI/IP limits and ensures reliable networking. However, you can override this limit by customizing the node's kubelet configuration if needed.
2. System Pods and Overhead
   EKS nodes host system pods such as kube-proxy, coredns, and others. These pods consume resources and reduce the available capacity for workload pods.

Default Pod Limits by Instance Type

Below is a table showing default pod limits for common EC2 instance types in EKS:

Instance TypeMax ENIsIPs per ENIMax Pods

How to Check Pod Capacity on an EKS Node

To determine the maximum pod capacity of your nodes, run the following command:

kubectl get nodes -o custom-columns="NAME:.metadata.name,PODS_CAPACITY:.status.allocatable.pods"

This will show the allocatable pods for each node, considering both ENI/IP limits and kubelet settings.

Customizing Pod Limits

If you need to override the default max-pods value, follow these steps:

1. Modify the Bootstrap Script

• EKS uses a bootstrap script to configure the node during startup. You can pass custom parameters to adjust the max-pods value.
• For example, add the following parameter when launching your worker nodes:

/etc/eks/bootstrap.sh <cluster-name> --use-max-pods false --kubelet-extra-args "--max-pods=<desired-pod-limit>"

1. Use a Custom AMI

• Create a custom AMI with your own kubelet configuration file. This method is more complex but provides greater control.

1. Monitor Pod Usage

• Use CloudWatch or Kubernetes metrics to monitor pod usage and ensure nodes are not over-provisioned or under-utilized.

Monitoring and Alerting

To avoid overloading your nodes, consider setting up alerts for high pod usage or resource exhaustion:

1. Use AWS CloudWatch Alarms to monitor node memory, CPU, and disk usage.
2. Configure Kubernetes Horizontal Pod Autoscalers (HPA) or Cluster Autoscalers to scale your workload or cluster as needed.

Conclusion

The number of pods that can fit on an AWS EKS node depends on ENI/IP limits, instance resources, and Kubernetes configurations. While AWS provides default limits, you can customize them to suit your needs. Understanding and monitoring these limits helps you maximize performance and avoid disruptions.

References:

1. AWS EKS Documentation: Networking Basics
2. AWS EC2 Instance Limits.
3. How many pods fit on an AWS EKS node?