Kubernetes Networking

What is Kubernetes Networking?

Since a Kubernetes cluster consists of various nodes and pods, understanding how they communicate between them is essential. The Kubernetes networking model supports different types of open source implementations. Kubernetes provides an IP address to each pod so that there is no need to map host ports to container ports as in the Docker networking model. Pods behave much like VMs or physical hosts with respect to port allocation, naming, load balancing and application configuration. For more background on Kubernetes components, see Kubernetes Architecture.

For further reading, see Kubernetes Documentation: Cluster Networking – Summary ›

Kubernetes vs. Docker Networking Model

The Docker networking model relies, by default, on a virtual bridge network called Docker0. It is a per-host private network where containers get attached (and thus can reach each other) and allocated a private IP address. This means containers running on different machines are not able to communicate with each other (as they are attached to different hosts’ networks). In order to communicate across nodes with Docker, we have to map host ports to container ports and proxy the traffic. In this scenario, it’s up to the Docker operator to avoid port clashes between containers.

The Kubernetes networking model, on the other hand, natively supports multi-host networking in which pods are able to communicate with each other by default, regardless of which host they live in. Kubernetes does not provide an implementation of this model by default, rather it relies on third-party tools that comply with the following requirements: all containers are able to communicate with each other without NAT; nodes are able to communicate with containers without NAT; and a container’s IP address is the same from inside and outside the container.

Kubernetes follows an “IP-per-pod” model where each pod get assigned an IP address and all containers in a single pod share the same network namespaces and IP address. Containers in the same pod can therefore reach each other’s ports via localhost:<port>. However, it is not recommended to communicate directly with a pod via its IP address due to pod’s volatility (a pod can be killed and replaced at any moment). Instead, use a Kubernetes service which represents a group of pods acting as a single entity to the outside. Services get allocated their own IP address in the cluster and provide a reliable entry point.

For further reading, see Kubernetes Documentation: Cluster Networking – Kubernetes Model ›

Kubernetes Networking Model Implementations

Kubernetes does not provide a default network implementation, it only enforces a model for third-party tools to implement. There is a variety of implementations nowadays, below we list some popular ones.

• Flannel – a very simple overlay network that satisfies the Kubernetes requirements. Flannel runs an agent on each host and allocates a subnet lease to each of them out of a larger, preconfigured address space. Flannel creates a flat network called as overlay network which runs above the host network.
• Project Calico – an open source container networking provider and network policy engine. Calico provides a highly scalable networking and network policy solution for connecting Kubernetes pods based on the same IP networking principles as the internet. Calico can be deployed without encapsulation or overlays to provide high-performance, high-scale data center networking.

Other options include Cisco ACI , Cilium , Contiv , Contrail , Kube-router , Nuage , OVN , Romana , VMWare NSX-T with NSX-T Container Plug-in (NCP) . Some tools even support using multiple implementations, such as Huawei CNI-Genie and Multus.

For further reading, see Kubernetes Documentation: Cluster Networking – How to achieve this ›

How Pods Communicate with Each Other

Because each pod has a unique IP in a flat address space inside the Kubernetes cluster, direct pod-to-pod communication is possible without requiring any kind of proxy or address translation. This also allows using standard ports for most applications as there is no need to route traffic from a host port to a container port, as in Docker. Note that because all containers in a pod share the same IP address, container-private ports are not possible (containers can access each other’s ports via localhost:<port>) and port conflicts are possible. However, the typical use case for a pod is to run a single application service (in a similar fashion to a VM), in which case port conflicts are a rare situation.

How Pods Communicate with Services

Kubernetes services allow grouping pods under a common access policy (for example, load-balanced). The service gets assigned a virtual IP which pods outside the service can communicate with. Those requests are then transparently proxied (via the kube-proxy component that runs on each node) to the pods inside the service. Different proxy-modes are supported:

• iptables: kube-proxy installs iptables rules trap access to service IP addresses and redirect them to the correct pods.
• userspace: kube-proxy opens a port (randomly chosen) on the local node. Requests on this “proxy port” get proxied to one of the service’s pods (as retrieved from Endpoints API).
• ipvs (from Kubernetes 1.9): calls netlink interface to create ipvs rules and regularly synchronizes them with the Endpoints API.

Kubernetes also offers an Endpoints API for Kubernetes native applications that is updated whenever the set of pods in a service changes. This allows a pod to retrieve the current endpoints for all pods in a service.

For further reading, see community documentation: Services – Virtual IPs and service proxies ›

Incoming Traffic from the Outside World

Nodes inside a Kubernetes cluster are firewalled from the Internet by default, thus services IP addresses are only targetable within the cluster network. In order to allow incoming traffic from outside the cluster, a service specification can map the service to one or more externalIPs (external to the cluster). Requests arriving at an external IP address get routed by the underlying cloud provider to a node in the cluster (usually via a load balancer outside Kubernetes). The node then knows which service is mapped to the external IP and also which pods are part of the service, thus routing the request to an appropriate pod.

To support more complex policies on incoming traffic, Kubernetes provides an Ingress API offering externally-reachable URLs, traffic load balancing, SSL termination, and name based virtual hosting to services. An ingress is a collection of rules that allow inbound connections to reach the cluster service. Note that to actually action ingresses specified via the API, an ingress controller (such as the NGINX ingress controller) must be deployed and configured for the cluster. This might be done automatically or not, depending on which Kubernetes cloud provider you are using.

A minimal ingress specification might look like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
backend:
serviceName: my-service
servicePort: 80

When processing this specification, the ingress controller would allocate an external IP to satisfy my-ingress rules, and forward all requests arriving at that IP to my-service:80 .

A slightly more complex example using simple load balancing could look like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /foo
        backend:
          serviceName: my-service-1
          servicePort: 80
      - path: /bar
        backend:
          serviceName: my-service-2
          servicePort: 80

In this case all requests sent to foo.bar.com/foo get routed (through node load balancing) to my-service-1:80 while requests sent to foo.bar.com/bar get routed (through node load balancing) to my-service-2:80

Note that the load balancing strategy is defined by the ingress controller and applied to all ingresses, thus does not appear in the ingress specification.

For further reading, see Kubernetes documentation: Ingress ›

DNS for Services and Pods

Kubernetes provides its own DNS service to resolve domain names inside the cluster in order for pods to communicate with each other. This is implemented by deploying a regular Kubernetes service which does name resolution inside the cluster, and configuring individual containers to contact the DNS service to resolve domain names. Note that this “internal DNS” is compatible and expected to run along with the cloud provider’s DNS service.

Every service gets assigned a DNS name which resolves to the cluster IP of the service. The naming convention includes the service name and its namespace. For example:

my-service.my-namespace.svc.cluster.local

A pod inside the same namespace as the service does not need to fully qualify its name, for example a pod in my-namespace could lookup this service with a DNS query for my-service , while a pod outside my-namespace would have to query for my-service.my-namespace.

For headless services (without a cluster IP), the DNS name resolves to the set of IPs of the pods which are part of the service. The caller can then use the set of IPs as it sees fit (for example round-robin).

By default pods get assigned a DNS name which includes the pod’s IP address and namespace. In order to assign a more meaningful DNS name, the pod’s specification can specify a hostname and subdomain:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  labels:
    name: busybox
spec:
  hostname: busybox
  subdomain: my-subdomain
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    name: busybox

In the above example the pod’s DNS name would be:

busybox.my-subdomain.svc.cluster.local

and resolve to the pod’s IP address.

Since Kubernetes 1.9, pods DNS settings can be customized in their specification. For example in this spec:

apiVersion: v1
kind: Pod
metadata:
  namespace: my-namespace
  name: my-pod-dns
spec:
  containers:
    - name: my-nginx
      image: nginx
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 1.2.3.4

Note that dnsPolicy set to None tells Kubernetes not to apply the default DNS settings, followed by the custom dnsConfig settings for this pod. In this example nameservers tell Kubernetes to use 1.2.3.4 as the DNS server address for this pod (a maximum of three servers can be specified in the list).

For further reading, see Kubernetes documentation: DNS for Services and Pods ›

Network Policies

By default pods can accept traffic from any pod in the cluster. Network policies can restrict how groups of pods are allowed to communicate between them and other network endpoints. A network policy specification uses labels to select pods and define a set of rules which govern what traffic is allowed to and from those pods. For example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: my-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - ipBlock:
        cidr: 172.17.0.0/16
        except:
        - 172.17.1.0/24
    - namespaceSelector:
        matchLabels:
          project: my-project
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 6379
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/24
    ports:
    - protocol: TCP
      port: 5978

podSelector specifies the group of pods on which this network policy should be applied. An empty podSelector would select all pods in the namespace. In this example only pods labelled role: db in namespace default are subject to this policy, which defines both Ingress and Egress rules.

The Ingress rules in this example specify that affected pods can only receive TCP traffic on port 6379 if originating from any of these sources:

• IP addresses in CIDR 172.17.0.0/16 but not in CIDR 172.17.1.0/24
• any pod in the namespace my-project
• any pod in the namespace default with label role: frontend

The Egress rules in this example specify that affected pods can only send TCP traffic to port 5978 of IP addresses in CIDR 10.0.0.0/24.

Besides whitelisting traffic sources/targets as in the above example, it is also possible to set default behaviors all incoming or outcoming traffic. For example to deny all incoming traffic to all pods in default namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Ingress

And to allow all outgoing traffic from pods in default namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-egress
  namespace: default
spec:
  podSelector: {}
  egress:
  - {}

Note that the network solution in use must support network policies for them to actually take effect. See Kubernetes Networking Model Implementations.

For further reading, see Kubernetes documentation: Network Policies ›

Network Extensions

Network extensions are the way to extend or enhance the networking by introducing various network functionalities such as cross-node networking or network policies. There are two types of network extensions or plugins:

• CNI plugin: designed for interoperability, this plugin implements the Container Network Interface (CNI) specification inserting a network interface into the container network namespace (e.g. one end of a veth pair) and making any necessary changes on the host (for example, attaching the other end of the veth into a bridge).
• Kubenet plugin: a simple plugin for Linux usage only, typically used together with a cloud provider that sets up routing rules for communication between nodes.

For further reading, see Kubernetes documentation: Network Plugins ›

Summary

Kubernetes provides a networking model but the specific implementation is done by a third party tool. By default, the model provides full communication inside the cluster (each pod gets its own IP address and can communicate with any other pod or service) and full isolation to the outside world. By specifying ingress rules, incoming outside traffic can be routed to services running in the cluster. Also, network policies specifications allow defining both ingress and egress rules for a set of pods (tagged with a given label) within a namespace. Such rules are defined based, for example, on IP address, namespace or pod labels.

Top Kubernetes Networking Tutorials from the Community

Zen and the Art of Kubernetes Networking

Tutorial by: Mei Long

Length: Short

Can help you learn: Basics of Kubernetes networking components

Tutorial steps:

1. Understanding Kubernetes networking components.
2. Creating sample deployments and port mappings.
3. Creating and exposing NodePort and ClusterIP type of services.

Kubernetes Networking Videos