IP Exhaustion Crisis in AKS Networking (Kubenet vs. Azure CNI)

Day 2: Solving the IP Exhaustion Crisis in AKS Networking

Welcome back to my Azure Kubernetes Service (AKS) learning journey! Today, I tackled one of the most confusing (and critical) topics in Kubernetes: Networking.

Specifically, I dove into the battle between the two main networking models—Kubenet and Azure CNI—and discovered a "best of both worlds" feature called Azure CNI Overlay.

If you've ever worried about running out of IP addresses in your virtual network, this post is for you.

Great IP Math Problem

To understand why networking matters, I looked at a practical exercise. Imagine we have a specific Subnet with a CIDR of /24.

A /24 subnet has 256 IP addresses. However, Azure always reserves the first few IPs for management (specifically 5 IPs), leaving us with 251 usable IP addresses.

Let's see how many resources we can deploy in this subnet using the two different models.

1. The Kubenet Approach (The Saver)

In Kubenet:

Nodes get IP addresses from the Azure Subnet.
Pods get IP addresses from a separate, internal "Pod CIDR" (not the subnet).

The Math:

Since only nodes need subnet IPs, we can deploy up to 251 Nodes.

If we multiply that by the default maximum pods per node (110), we can run:

251 nodes x 110 pods = 27,610 pods{alertSuccess}

251 \text{ nodes} \times 110 \text{ pods} = \mathbf{27,610 \text{ pods}}

Kubenet is incredibly efficient with IP space.

2. Azure CNI Approach (The Spender)

In the traditional Azure CNI (Container Networking Interface):

Nodes get IP addresses from the Azure Subnet.
Pods ALSO get IP addresses from the Azure Subnet.

The Math:

Every single pod eats up a real IP from your VNet. If we use the default limit of 30 pods per node:

We can only deploy about 8 Nodes.
Total pods = 8 nodes × 30 pods = 240 Pods.

We hit an IP exhaustion wall very quickly. We went from potentially 27,000 pods (Kubenet) to just 240 (Azure CNI) in the same subnet size!

So, Which One Should I Choose?

Based on the math above, here is the decision matrix I learned:

Use Kubenet when:

You have limited IP address space.
Most communication happens inside the cluster.
You don't need advanced features like Virtual Nodes or Azure Network Policies.

Use Azure CNI when:

You have plenty of IP addresses available.
Pods need to speak directly to resources outside the cluster (like VMs, on-prem networks, or Azure SQL endpoints) without NAT.
You need advanced features like Virtual Nodes or Azure Network Policies.
You want to avoid managing User Defined Routes (UDRs) manually.

Solution: Azure CNI Overlay

What if we want the performance and features of Azure CNI, but the IP efficiency of Kubenet?

Enter Azure CNI Overlay.

This was the highlight of my learning today. It's a mode that combines the best of both worlds.

How it works

Nodes take IPs from your Subnet (just like before).
Pods take IPs from a private Overlay Network (a separate CIDR you define during creation).
Each node gets assigned a large slice (a /24) of that overlay network for its pods.

Why is this huge?

No IP Exhaustion: Pods don't consume your VNet IPs.
No Route Tables: Unlike Kubenet, you don't need to mess around with User Defined Routes (UDRs) or Route Tables.
Performance: It promises great connectivity performance without requiring extra hops or encapsulation methods usually needed to tunnel traffic.

Note: Because the overlay network is isolated, external endpoints cannot connect directly to a specific Pod IP. But for most use cases, this is fine.

Kubenet vs. Azure CNI Overlay: Ultimate Showdown

Since both Kubenet and CNI Overlay save IP addresses, how do they compare against each other? It turns out Azure CNI Overlay is vastly superior for modern workloads. Here is what I discovered:

Max Scale: Kubenet limits you to 400 nodes per cluster because it relies on Azure Route Tables (which have a limit of 400 routes). CNI Overlay doesn't use these route tables, allowing you to scale up to 1,000 nodes per cluster!
Performance: Kubenet introduces minor latency because traffic has to make an extra hop through those route tables. CNI Overlay provides the best performance for pod-to-pod communication because that extra hop is removed.
Network Policies: Kubenet only supports Calico network policies. CNI Overlay supports Azure Network Policies, Calico, and Cilium.
OS Support: Kubenet is Linux-only. CNI Overlay supports both Linux and Windows Server 2022.
Complexity: Kubenet is complex because you have to correctly configure and manage Azure Route Tables on the cluster subnet. CNI Overlay is delightfully simple—no additional configuration is required for pod networking.

⚠️ Overlay Limitations to Keep in Mind

While it is a powerful feature, CNI Overlay isn't perfect yet. I learned about two key limitations:

You cannot use the Application Gateway Ingress Controller (AGIC). Standard Nginx ingress controllers work fine, but AGIC is not supported.
It does not support Windows Server 2019 node pools (only 2022).

Hands-On: Creating an Overlay Cluster

I decided to try this out using the Azure CLI. Here is how I spun up a cluster using the Overlay mode.

1. Command

We need three specific flags:

--network-plugin azure (Use CNI)
--network-plugin-mode overlay (Turn on Overlay)
--pod-cidr (Define the private range for pods)

Bash

# Create Resource Group
az group create -n rg-aks-cni-overlay -l westeurope

# Create AKS Cluster with Overlay
az aks create -n aks-cni-overlay -g rg-aks-cni-overlay \
    --network-plugin azure \
    --network-plugin-mode overlay \
    --pod-cidr 192.168.0.0/16

2. Exploring the Result

Once the cluster was ready, I poked around to see how the IPs were assigned.

Checking Nodes:

kubectl get nodes -o wide

Result: The nodes had IPs from the Azure Subnet range.

Checking Pods:

kubectl get pods -A -o wide

I noticed two patterns here:

Host Network Pods: System pods like kube-proxy or ip-masq-agent used the Node's IP address.
Overlay Pods: Pods like coredns or my nginx application pods used IPs from the 192.168.0.0/16 range I defined earlier.

When I deployed 10 sample Nginx pods, they all received 192.168.x.x addresses, confirming that my VNet IPs were safe and unconsumed!

What Exactly is a CNI?

Before today, I sort of thought "CNI" was just an Azure term. But it turns out CNI (Container Network Interface) is actually an open standard in the Kubernetes world.

Because it's an open standard, any provider or community can create their own implementation (plugin) to dictate how pods get IP addresses and talk to each other. Over the years, the community has built a ton of these plugins, including:

Calico (famous for network policies)
Flannel (known for being incredibly simple)
Weave Net (often just called Weave)
Canal (a mix of Flannel and Calico)

"Bring Your Own CNI" (BYO CNI) Feature

By default, when you deploy an AKS cluster, Azure automatically installs and configures Kubenet or Azure CNI for you.

But what if your organization already uses Flannel on-premises and wants the exact same setup in Azure? That's where BYO CNI comes in. When you use this feature, AKS provisions the Kubernetes cluster but skips installing a networking plugin. The cluster will sit there in a "Not Ready" state until you manually deploy the CNI of your choice.

Enter Cilium: Star of BYO CNI

While you can technically install various plugins, the BYO CNI feature in AKS is most heavily associated with deploying Cilium.

Cilium is an incredibly powerful, modern CNI that has been taking the Kubernetes world by storm. When deploying Cilium on AKS, you have two options:

Open-Source Version: Free and maintained by the community.
Enterprise Version: Comes with official enterprise support, which is usually a requirement for big companies running critical production workloads.

Secret Sauce: eBPF

The main reason people go out of their way to install Cilium is a technology called eBPF (Extended Berkeley Packet Filter).

Without getting too deep into the weeds, eBPF allows Cilium to run networking, security, and observability rules directly inside the Linux kernel. This means traffic doesn't have to bounce around through complex routing tables or proxy servers (like kube-proxy). It is lightning fast, highly secure, and provides deep visibility into what your pods are doing.

Key Takeaways

CNI is an open standard: Anyone can build a network plugin for Kubernetes (Flannel, Weave, Canal, etc.).
BYO CNI: You can tell AKS not to install a default network, allowing you to bring your own.
Cilium is a powerhouse: It uses advanced kernel technology (eBPF) to provide blazing-fast networking and security, and you can run it in AKS using the BYO CNI method.
Kubenet is great for saving IPs but requires Route Tables.
Traditional Azure CNI is powerful but eats IPs for breakfast.
Azure CNI Overlay solves the exhaustion problem by keeping Pod IPs internal, while removing the complexity of Route Tables.

It feels like Overlay is going to be the default standard for many clusters going forward.