{"id":562,"date":"2019-02-13T12:00:17","date_gmt":"2019-02-13T12:00:17","guid":{"rendered":"https:\/\/cloudmanagement.navisite.com\/?p=562"},"modified":"2020-01-08T04:22:05","modified_gmt":"2020-01-08T04:22:05","slug":"azure-kubernetes-service-aks-network-design","status":"publish","type":"post","link":"https:\/\/www.navisite.com\/blog\/azure-kubernetes-service-aks-network-design\/","title":{"rendered":"Azure Kubernetes Service (AKS): Kubenet Network Design"},"content":{"rendered":"

 
\n\"\"<\/a>
\n\"\"<\/p>\n

Overview<\/h2>\n

Managed Kubernetes simplifies deployment, management and operations of Kubernetes, and allows developers to take advantage of Kubernetes without worrying about the underlying plumbing to get it up running and freeing up developer time to focus on the applications. Different Cloud Providers are offering this service \u2013 for example Google Kubernetes\u00a0Engine (GKE), Amazon has Elastic Container Service for Kubernetes (EKS), Microsoft has Azure Kubernetes Service (AKS) etc..
\nThe focus of this blog is on Azure Kubernetes Services.\u00a0AKS makes it easy to deploy and manage containerized applications without container orchestration expertise. Azure handles the ongoing operations including provisioning, upgrading and scaling of resources\/nodes. Worker nodes are deployed as Azure Virtual Machines. Master nodes are completely managed by Azure. In short, AKS reduces the complexity and operational overhead of managing a Kubernetes cluster, by offloading much of that responsibility to Azure. Azure handles health monitoring and maintenance.
\nI’m really excited to start off this first 2019 blog series. This is Part-1 of the Azure Container Series,<\/strong> part of the Azure Management Services<\/a> Navisite offers. This blog walks you through a step-by-step process to create a public facing “Load Balancer” service type in AKS. Once the sample application is deployed we will do a deep dive into networking and traffic flow.
\nThese are the planned blogs in this series so stay tuned…<\/p>\n

    \n
  1. Azure Kubernetes Services (AKS) – Kubenet Network Design (Part 1)<\/li>\n
  2. Azure Container Registry<\/li>\n
  3. Azure Kubernetes Services (AKS) – Advanced Network Design with CNI (Part 2)<\/li>\n
  4. Custom Kubernetes Cluster on IaaS VMs in Azure using Flannel Overlay<\/li>\n
  5. AKS with Persistent volumes using Azure Disks and Azure Files<\/li>\n<\/ol>\n

    AKS Reference Architecture (Kubenetes Networking)<\/h2>\n

    Throughout the blog article we will reference the following architecture. It shows a 3-nodes Kubernetes cluster with basic Kubenet networking in a flat-routed topology. The master nodes are completely managed by Azure.
    \n    \"\"<\/a>
    \n<\/a><\/p>\n

    Kubernetes Service Architecture<\/h2>\n

    To simplify the network configuration for application workloads, Kubernetes uses\u00a0Services<\/em>\u00a0to logically group a set of pods together and expose your application for external network connectivity. There are three types of services, or ServiceTypes.<\/p>\n

      \n
    1. ClusterIP<\/li>\n
    2. NodePort<\/li>\n
    3. LoadBalancer<\/li>\n<\/ol>\n

      We will focus on the LoadBalancer service type. It leverages an External Azure Load balancer\u00a0 with a Public IP.
      \nFrom Microsoft documentation:
      \n

      \"\"<\/a>
      Source: Microsoft Documentation<\/figcaption><\/figure><\/p>\n

      Install Azure CLI and login to Azure<\/h2>\n

      Azure Kubernetes Service management can be done from a development VM as well as using Azure Cloud Shell.\u00a0 In my setup, I’m using an Ubuntu VM and I’ve install Azure CLI locally. To install Azure CLI follow this link<\/a>.
      \nFew basic commands to login to Azure using Azure CLI<\/p>\n

      az login\naz account set --subscription \"Microsoft Azure XXXX\"\naz account show --output table\n<\/pre>\n
      Create AKS Cluster and Connect to It<\/h2>\n
      Create the AKS cluster in Azure is a single command. In Azure, create a resource group to manage the AKS cluster resources first.<\/p>\n
      *********************\nOn the Development VM\n**********************\nnehali@nn-ubuntu-vm:~$ az group create --name nn-aks-rg --location eastus\n{\n\"id\": \"\/subscriptions\/XXXXXXX-f308-496c-a43c-faaeXXXXXX\/resourceGroups\/nn-aks-rg\",\n\"location\": \"eastus\",\n\"managedBy\": null,\n\"name\": \"nn-aks-rg\",\n\"properties\": {\n\"provisioningState\": \"Succeeded\"\n},\n\"tags\": null,\n\"type\": null\n}\nCreate AKS Cluster:\naz aks create \\\n--resource-group nn-aks-rg \\\n--name nn-aks-cluster \\\n--node-count 3 \\\n--enable-addons monitoring \\\n--generate-ssh-keys \\\n--node-vm-size Standard_DS1_v2 \\\n--dns-name-prefix nnakscluster\nInstall kubectl (If not installed already!)\nConnect to AKS Cluster from the development VM:\nnehali@nn-ubuntu-vm:~$ az aks get-credentials --resource-group nn-aks-rg --name nn-aks-cluster\nMerged \"nn-aks-cluster\" as current context in \/home\/nehali\/.kube\/config\nnehali@nn-ubuntu-vm:~$ kubectl config get-clusters\nNAME\nnn-aks-cluster\nnehali@nn-ubuntu-vm:~$ kubectl config current-context\nnn-aks-cluster\nnehali@nn-ubuntu-vm:~$ kubectl config current-context\nnn-aks-cluster\nnehali@nn-ubuntu-vm:~$ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\naks-nodepool1-19416140-0 Ready agent 14m v1.9.11\naks-nodepool1-19416140-1 Ready agent 14m v1.9.11\naks-nodepool1-19416140-2 Ready agent 14m v1.9.11\nNote down the nodes IP from the output below\nnehali@nn-ubuntu-vm:~$ kubectl get nodes -o wide\nNAME                       STATUS   ROLES   AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME\naks-nodepool1-19416140-0   Ready    agent   23h   v1.9.11   10.240.0.4    <none>        Ubuntu 16.04.5 LTS   4.15.0-1035-azure   docker:\/\/3.0.1\naks-nodepool1-19416140-1   Ready    agent   23h   v1.9.11   10.240.0.5    <none>        Ubuntu 16.04.5 LTS   4.15.0-1035-azure   docker:\/\/3.0.1\naks-nodepool1-19416140-2   Ready    agent   23h   v1.9.11   10.240.0.6    <none>        Ubuntu 16.04.5 LTS   4.15.0-1035-azure   docker:\/\/3.0.1\nnehali@nn-ubuntu-vm:~\/.ssh$ kubectl version\nClient Version: version.Info{Major:\"1\", Minor:\"13\", GitVersion:\"v1.13.2\", GitCommit:\"cff46ab41ff0bb44d8584413b598ad8360ec1\ndef\", GitTreeState:\"clean\", BuildDate:\"2019-01-10T23:35:51Z\", GoVersion:\"go1.11.4\", Compiler:\"gc\", Platform:\"linux\/amd64\"}\nServer Version: version.Info{Major:\"1\", Minor:\"9\", GitVersion:\"v1.9.11\", GitCommit:\"1bfeeb6f212135a22dc787b73e1980e5bccef1\n3d\", GitTreeState:\"clean\", BuildDate:\"2018-09-28T21:35:22Z\", GoVersion:\"go1.9.3\", Compiler:\"gc\", Platform:\"linux\/amd64\"}<\/pre>\n
      Validations in Azure<\/h2>\n
      Once the Azure Kubernetes Service Cluster is created, login to the Azure Portal and verify the Resource Groups, Service Principal, three nodes IPs and the Route table for the inter pod routing.<\/p>\n
      Resource Groups<\/strong><\/h5>\n

      \n\"\"<\/a><\/p>\n
      Service Principal\"\"<\/a><\/h5>\n
      Kubernetes Nodes<\/h5>\n

      \n\"\"<\/a>
      \nRoute Table<\/strong>
      \n\"\"<\/a><\/p>\n
      Load Balancer<\/h5>\n

      \n\"\"<\/a><\/p>\n
      Run a Sample Containerized Application<\/h2>\n
      Deployment Manifest file<\/h5>\n
      Create a Kubernetes manifest file for the deployment. A deployment in Kubernetes represents one or more identical pods that are managed by Kubernetes deployment controller.\u00a0 It also defines the number of replica sets (pods) to create. In our case we create a file called nn-deployment.yaml which uses the nginx container image and 3 replicas.\u00a0 We will use a separate manifest file for services.<\/p>\n
      nehali@nn-ubuntu-vm:~$ more nn-deployment.yaml\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\nname: nn-nginx-deployment\nlabels:\napp: nn-nginx\nspec:\nreplicas: 3\nselector:\nmatchLabels:\napp: nn-nginx\ntemplate:\nmetadata:\nlabels:\napp: nn-nginx\nspec:\ncontainers:\n- name: nnc-nginx\nimage: nginxdemos\/hello\nports:\n- containerPort: 80\nnehali@nn-ubuntu-vm:~$ kubectl create -f nn-deployment.yaml\ndeployment.apps\/nn-nginx-deployment created\nnehali@nn-ubuntu-vm:~$ kubectl get deployments\nNAME READY UP-TO-DATE AVAILABLE AGE\nnn-nginx-deployment 3\/3 3 3 43m\nnehali@nn-ubuntu-vm:~$ kubectl get pods\nNAME                                  READY   STATUS    RESTARTS   AGE\naks-ssh-6fbc77d848-ghdzh              1\/1     Running   6          22h\nnn-nginx-deployment-77fcff4b8-f6pxc   1\/1     Running   0          20h\nnn-nginx-deployment-77fcff4b8-klvsj   1\/1     Running   0          20h\nnn-nginx-deployment-77fcff4b8-n98q9   1\/1     Running   0          20h\nGet the POD IPs using the -o wide switch:\nnehali@nn-ubuntu-vm:~$ kubectl get pods -o wide\nNAME                                  READY   STATUS    RESTARTS   AGE   IP           NODE                       NOMINATED NODE   READINESS GATES\naks-ssh-6fbc77d848-ghdzh              1\/1     Running   6          22h   10.244.0.7   aks-nodepool1-19416140-2   <none>           <none>\nnn-nginx-deployment-77fcff4b8-f6pxc   1\/1     Running   0          20h   10.244.2.9   aks-nodepool1-19416140-1   <none>           <none>\nnn-nginx-deployment-77fcff4b8-klvsj   1\/1     Running   0          20h   10.244.0.9   aks-nodepool1-19416140-2   <none>           <none>\nnn-nginx-deployment-77fcff4b8-n98q9   1\/1     Running   0          20h   10.244.1.9   aks-nodepool1-19416140-0   <none>           <none>\nnehali@nn-ubuntu-vm:~$ kubectl get pods -o yaml | grep -i PODIP\n    podIP: 10.244.0.7\n    podIP: 10.244.2.9\n    podIP: 10.244.0.9\n    podIP: 10.244.1.9\n<\/pre>\n
      Service Manifest file<\/h5>\n
      Azure Kubernetes uses Services to logically group a set of pods together and provide network connectivity. As explained in the architecture section, there are three types of services<\/a>. In this example, we will use the LoadBalancer<\/em> service type. The following manifest file creates an external public IP address and connects the requested pods to the load balancer pool.<\/p>\n
      nehali@nn-ubuntu-vm:~$ more nn-service.yaml\napiVersion: v1\nkind: Service\nmetadata:\n  name: nn-nginx-service\nspec:\n  type: LoadBalancer\n  ports:\n  - port: 80\n  selector:\n    app: nn-nginx\nnehali@nn-ubuntu-vm:~$ kubectl create -f nn-service.yaml\nservice\/nn-nginx-service created\nnehali@nn-ubuntu-vm:~$ kubectl get service\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nkubernetes ClusterIP 10.0.0.1 <none> 443\/TCP 3h55m\nnn-nginx-service LoadBalancer 10.0.121.81 <pending> 80:32210\/TCP 24s\nnehali@nn-ubuntu-vm:~$ kubectl get service --watch\nNote the Private and Public IPs for the service and the corresponding POD endpoints\nnehali@nn-ubuntu-vm:~$ kubectl get service\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nkubernetes ClusterIP 10.0.0.1 <none> 443\/TCP 6h37m\nnn-nginx-service LoadBalancer 10.0.140.61 40.71.30.139 80:31278\/TCP 148m\nnehali@nn-ubuntu-vm:~$ kubectl get endpoints nn-nginx-service\nNAME ENDPOINTS AGE\nnn-nginx-service 10.244.0.9:80,10.244.1.9:80,10.244.2.9:80 149m<\/pre>\n
       <\/p>\n
      SSH into the AKS Nodes<\/h2>\n
      Throughout the lifecycle of your Azure Kubernetes Service cluster, you may need to access an AKS node. This access could be for maintenance, log collection, or other troubleshooting operations. The AKS nodes are Linux VMs, so you can access them using SSH. For security purposes, the AKS nodes are not exposed to the internet and master nodes are fully managed by Azure.
      \nThis article shows you how to create an SSH connection with an AKS node using their private IP addresses. Detailed documentation       here<\/a>.<\/p>\n
      Get the resource Group:\nnehali@nn-ubuntu-vm:~$ az aks show --resource-group nn-aks-rg --name nn-aks-cluster --query nodeResourceGroup -o tsv\nMC_nn-aks-rg_nn-aks-cluster_eastus\nGet the list of VMs\nnehali@nn-ubuntu-vm:~$ az vm list --resource-group MC_nn-aks-rg_nn-aks-cluster_eastus -o table\nName ResourceGroup Location Zones\n------------------------ ---------------------------------- ---------- -------\naks-nodepool1-19416140-0 MC_nn-aks-rg_nn-aks-cluster_eastus eastus\naks-nodepool1-19416140-1 MC_nn-aks-rg_nn-aks-cluster_eastus eastus\naks-nodepool1-19416140-2 MC_nn-aks-rg_nn-aks-cluster_eastus eastus\nAdd the public key to the nodes\naz vm user update \\\n--resource-group MC_nn-aks-rg_nn-aks-cluster_eastus \\\n--name aks-nodepool1-19416140-0 \\\n--username azureuser \\\n--ssh-key-value ~\/.ssh\/id_rsa.pub\naz vm user update \\\n--resource-group MC_nn-aks-rg_nn-aks-cluster_eastus \\\n--name aks-nodepool1-19416140-1 \\\n--username azureuser \\\n--ssh-key-value ~\/.ssh\/id_rsa.pub\naz vm user update \\\n--resource-group MC_nn-aks-rg_nn-aks-cluster_eastus \\\n--name aks-nodepool1-19416140-2 \\\n--username azureuser \\\n--ssh-key-value ~\/.ssh\/id_rsa.pub\nGet the list of node IPs:\nnehali@nn-ubuntu-vm:~$ az vm list-ip-addresses --resource-group MC_nn-aks-rg_nn-aks-cluster_eastus -o table\nVirtualMachine PrivateIPAddresses\n------------------------ --------------------\naks-nodepool1-19416140-0 10.240.0.4\naks-nodepool1-19416140-1 10.240.0.5\naks-nodepool1-19416140-2 10.240.0.6\nRun an ubuntu container image and attach a terminal session to it. We will use this container to ssh to any of the AKS cluster nodes.\nkubectl run -it --rm aks-ssh --image=ubuntu\napt-get update && apt-get install openssh-client -y\nIn a Seperate window\nnehali@nn-ubuntu-vm:~$ kubectl get pods\nNAME READY STATUS RESTARTS AGE\naks-ssh-6fbc77d848-h52wc 1\/1 Running 0 43s\nnn-nginx-deployment-7489bc85cf-95jxn 1\/1 Running 0 15m\nnn-nginx-deployment-7489bc85cf-xwllg 1\/1 Running 0 15m\nnn-nginx-deployment-7489bc85cf-zp68z 1\/1 Running 0 15m\nCopy the ssh private key to the newly deployed pod.\nnehali@nn-ubuntu-vm:~$ kubectl cp ~\/.ssh\/id_rsa aks-ssh-6fbc77d848-h52wc:\/id_rsa\nBack in the container terminal\nroot@aks-ssh-6fbc77d848-h52wc:\/# ls\nbin boot dev etc home id_rsa lib lib64 media mnt opt proc root run sbin srv sys tmp usr var\nroot@aks-ssh-6fbc77d848-h52wc:\/# chmod 0600 id_rsa\nroot@aks-ssh-6fbc77d848-h52wc:\/# mv id_rsa ~\/.ssh\/\nroot@aks-ssh-6fbc77d848-h52wc:~# cd .ssh\/\nroot@aks-ssh-6fbc77d848-h52wc:~\/.ssh# ls\nid_rsa known_hosts\nFrom here on you can ssh to any of the AKS nodes.\nroot@aks-ssh-6fbc77d848-h52wc:~\/.ssh# ssh azureuser@10.240.0.4\nYay!<\/pre>\n
      Inspect Kubenetes Networking<\/h2>\n
      In Azure Kubernetes Service, you can deploy a cluster that uses one of the following two network models:<\/p>\n