The code review and content of CKA exam...

Need for a container orchestration tool

• Trend from Monolith to Microservice
• Increase usage of containers
• Deman for a proper way of managing those hundreds of containers

• High Availability or no downtime
• Scalability or high performance
• Disaster recovery - Backup and restore

Worker Machine in K8s Cluster

• Each Node has multiple Pods on it
• Worker Nodes do the actual work
• 3 Processes must be installed on every Node
  • Container runtime
  • Kubelet
    • Kubelet interacts with both the container and node
    • Kubelet starts the Pod with a container inside
  • Kube Proxy
    • Kube Proxy forwards the requests

So, how do you interact with this cluster?

• How to:
  • Scheduler Pods?
  • Monitor?
  • re-scheduled/Re-start Pod
  • Join a new Worker?
  • etc
• Managing processes are done by Master Nodes (The Control Plane)

• 4 processes runs on every control plane node
  • Api Server
    • Cluster gateway
    • Acts as the gatekeeper for authentication!
    • Only 1 entrypoint into the cluster
    • Api Server is load balanced
  • Scheduler
    • Scheduler: Where to put the Pod?
    • Scheduler just decides on which Node new Pod should be scheduled?
    • Kubelet actually starts the Pod
  • Controller Manager
    • Controller Manager: Detects cluster state changes
  • etcd
    • Etcd is the cluster brain
    • Cluster changes get stored here
    • Application data is NOT stored in etcd!
    • Distributed storage across all master nodes

• Very powerful way that enables interaction with the cluster
• Limitation when using those commands
• Best practice: Use K8s Configuration file

• Write the component configuration in a file
• Apply configuration file with kubectl
  • Multiple K8s components in 1 file and apply them with 1 apply command
• Update K8s components:
  • Edit the config file and apply it again
• Delete K8s components with config file

• Imperative
  • Telling Kubernetes WHAT to do
  • We operate directly on live objects
• Declarative
  • Telling Kubernetes WHAT we want as the end result in the config file
  • We operate on object configuration files
  • We don't define the operations, these are automatically detected by kubectl

Which one to use?

• Imperative
  • Practical when testing
  • Or for quick one-off tasks
  • Or when just getting started
• Declarative
  • History of configurations
  • Infrastructure as Code in Git Repo
  • Collaboration and review processes possible
  • More transparent

• On Control Plane & Worker
  • Container Runtime (Run as regular Linux process)
  • Kubelet (Run as regular Linux process)
  • Kube Proxy (Pod)
• Only on Control Plane
  • Api Server (Pod)
  • Scheduler (Pod)
  • Controller Manger (Pod)
  • ETCD (Pod)

• Master components deployed as Pods
• Pods are deployed by master components
  • Send a request to API Server
  • Scheduler decides where to place Pod
  • Pod data stored in etcd store
• How to schedule the Master Pods then? (The Egg and Chicken Problem!)

Static Pods

• Are managed directly by the kubelet daemon
• Without control plane

• Regular Pod Scheduling
  • API Server gets the request
  • Scheduler: which Node?
  • Kubelet: schedules Pod
• Static Pod Scheduling
  • Kubelet: schedules Pod

How does that work?

• Kubelet watches a specific location on the Node it is running
  • /etc/kubernetes/manifests
• Schedules Pod, when it finds a "Pod" manifest

• Why is it called static Pod?

• How is it different?

• Kubelet (NOT Controller Manager) watches static Pods and restarts them if they fail

• Pod names are suffixed with the node hostname

• First step when installing K8s cluster

  • Generate static Pods manifests
  • Put those config files into the correct folder

Everything needs a certificate...

How does it work?

• Generate self-signed CA certificate for Kubernetes (cluster root CA)
• Sign all client and server certificates with it
• Certificates are stored in: /etc/kubernetes/pki
• Each component gets a certificate, signed by the same certificate authority
• Proof that components identify and that its part of the same cluster

1. Generate a self-signed CA certificate for the whole Kubernetes cluster (cluster root CA)
2. Sign all client and server certificates with it
   • Server certificate for the API server endpoint
   • Client certificate for scheduler and controller manager
   • Server certificate for Etcd and Kubelet
   • Client certificate for API Server to talk to Kubelet and Etcd
   • Client certificate for Kubelet to authenticate to API Server

Public Key Infrastructure

• Governs the issuance of certificates to:
  • Protect sensitive data
  • Provide unique digital identities for applications, users and devices
  • Secure end-to-end communication

For a K8s cluster, we need to do all the steps above + some other configuration details we need to provide

But it is complex and time consuming, when doing it manually

• Kubeadm

  • Toolkit for bootstrapping a best-practices K8s cluster

• Providing fast paths for creating K8s cluster

• Performs the actions necessary to get a minimum viable cluster

• It cares only about bootstrapping, not about provisioning machines

• Maintained by Kubernetes

• All

  sudo swapoff -a

• All

  sudo vim /etc/hosts

• Add all the server ips and correspond names in /etc/hosts file, e.g.

  172.31.44.88 master
  172.31.44.219 worker1
  172.31.37.5 worker2
  

• All

  sudo hostnamectl set-hostname <correspond names e.g. master>

• All

  cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
  overlay
  br_netfilter
  EOF
  sudo modprobe overlay
  sudo modprobe br_netfilter
  cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
  net.bridge.bridge-nf-call-iptables  = 1
  net.bridge.bridge-nf-call-ip6tables = 1
  net.ipv4.ip_forward                 = 1
  EOF
  sudo sysctl --system

• All

  sudo apt update
  sudo apt install -y containerd
  sudo mkdir -p /etc/containerd
  containerd config default | sudo tee /etc/containerd/config.toml
  sudo systemctl restart containerd
  service containerd status

• Kubelet

  • Does things like starting pods and containers
  • Components that runs on all the machines in your cluster

• Kubeadm

  • Command line tool to initialize the cluster

• Kubectl

  • Command line tool to talk to the cluster

• All

  sudo apt-get update
  sudo apt-get install -y apt-transport-https ca-certificates curl
  sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
  echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

NOTE-1: Kubelet, Kubeadm and Kubectl MOST be ALL in SAME VERSION...
NOTE-2: For seeing all available versions, use:

Use the apt-cache madison kubeadm to get started.

• All

  sudo apt-get update
  sudo apt-get install -y kubelet=<VERSION> kubeadm=<VERSION> kubectl=<VERSION>
  sudo apt-mark hold kubelet kubeadm kubectl

1. preflight
Checks to validate the system state making any changes
2. certs
Generate a self-signed CA to set up identities for each component in the cluster
3. kubeconfig
writes kubeconfig files in `/etc/kubernetes`

• Master

  sudo kubeadm init

• /etc/kubernetes
• /etc/kubernetes/manifests/*
• /var/lib/kubelet
• /var/lib/kubelet/pki
• /var/lib/kubelet/config.yaml

• Now we only can interact with kubectl this way:

  sudo kubectl get nodes --kubeconfig=/etc/kubernetes/admin.conf

But it's not an efficient way. So we have two options:

1. File passed with --kubeconfig flag
2. KUBECONFIG environment variable
3. File located in $HOME/.kube/config folder

Option 1 is not efficient because everytime you should pass this flag...
Option 2 is not efficient too because its only works in the current session...
But the option number 3 is awesome and actually a bets practise...

• Create .kube folder

  mkdir -p ~/.kube

• Copy admin.conf to this folder

  sudo cp -i /etc/kubernetes/admin.conf ~/.kube/config

• Change owner of this file to ourselves

  sudo chown $(id -u):$(id -g) ~/.kube/config

• Now everything is good, and we don't have to use sudo or --kubeconfig

  kubectl get nodes

• Why is a Pod abstraction useful?
• Container vs. Pod
• When are multiple containers necessary?
• How containers communicate in a Pod?

Every Pod has a unique IP address.
IP address reachable from all other Pods in K8s cluster.

• Container Port Mapping WITHOUT Pods
  Bind host port to application port in container (5432:5432)

• Own IP address
• Own Network namespace
• Virtual Ethernet Connection
• Pod is a Host
  • No conflicts...

If you change the container runtime in k8s, e.g. from containerd to docker, K8s configuration would stay the same...

Multiple containers in a Pod

• Hepler or sider application to your main application
• Called side-car containers

• containers can talk via localhost and port

• Pause container in each Pod
• Also called sandbox containers
• Reserve and holds network namespace (nets)
• Enables communication between containers

Networking WITHIIN PODS

• No built-in solution
• Expects you to implement a networking solution
• But impose fundamental requirements on any implementation to be pluggable into kubernetes

1. Every Pod gets its Own unique IP address
2. pods on same Node can Communicate with that IP address
3. pods on different Node can Communicate with that IP address without NAT(Network Address Translation)

• k8s doesn't care about the exact IP address

Many networking solutions, which implement this module

1. flannel
2. waveworks
3. cilium
4. VMware NSX

• Each Node gets an IP address from IP range from VPC

• Pods are isolated with own private network

• On each Node a private network with a different IP range is created (Bridge via a CNI plugin e.g. wavework)
  • IP address ranges should not overlap!
• Bridge enables Pod communication on the same Node

How do we make sure each Node gets a different set of IP addresses? We need to ensure unique IP's --> Because K8S doesn't care!

• CNI Plugin! (e.g. Wavework)
• Each Node gets an equal subset of this IP range
• Virtual Private Networks with own sets of IP addresses

• They can't talk directly, because of private isolated networks
• Pods can communicate via Gateways
• Network Plugin creates 1 large Pod Network
  1. Why? All Nodes are in the same network
  2. Can talk directly via their IPs
• Each Node can access via the virtual pod network on its Node
• K8s requirements for CNI Plugins
  1. Every Pod gets its own unique IP address
  2. Pods on same Node can communicate with that IP address
  3. Pods on different Node can communicate with that IP address with out NAT (Network Address translation)

So how to manage thousands of Nodes? we need a more Automated & Scalable solution...

• CNI Plugins solved this!

• A Weave network consists of peers - Weave Net routers reside on the Nodes
• They form a group and can directly talk to each other

Now is we should do something about master node status, because currently it is in NotReady statement... also the coredns pods can't be Ready because of our node statement... so we should implement a networking solution to achieve these goals... the solution is implenting a CNI Plugin... In this scenario, we are going with Weave Net

• Note: ONLY after this solution we can add Worker Nodes to our cluster...

Pod Network IP Address Range should not overlap with Node IP Address Range --> VPC IP != Nodes IP

• the default range that Weave Net would like to use is 10.32.0.0/12 - a 12-bit prefix, where all addresses start with the bit pattern 000010100010, or in decimal everything from 10.32.0.0 through 10.47.255.255.

• Before installing Weave Net, you should make sure the following ports are not blocked by your firewall: TCP 6783 and UDP 6783/6784. For more details, see the docs.

• Master

  wget "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" -O weave.yaml

• You can change the default IP range:

• Master

  vim weave.yaml
  # Then find:
  # containers:
  #  - name: weave
  #    command:
  #      - /home/weave/launch.sh
  # And add e.g. `- --ipalloc-range=100.32.0.0/12` bellow first command...

• Master

  kubectl apply -f weave.yaml

Be sure to check the IP address of Pods and Node in kube-system namespace...

• docs
• configuring-weave

We already installed:

• containerd

• kubeadm

• kubelet

• kubectl

• Now lets joining the worker nodes...

• NOTE: A bidirectional trust needs to be established:

  1. Discovery (Node trust the K8s Control Plane)
  2. TLS bootstrap (K8s Control Plane trust Node)

• master

  kubeadm token create --print-join-command

Paste the output of this command on Worker Node(s). Note that you can use this output as much as you want.

kubectl logs -n kube-system pod/weave-net-4xtwz -c weave

Now let's deploy an Nginx Deployment with 2 Pods and a test Service for it

• Nginx Deployment

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: nginx-deployment
    labels:
      app: nginx
  spec:
    replicas: 2
    selector:
      matchLabels:
        app: nginx
    template:
      metadata:
        labels:
          app: nginx
      spec:
        containers:
        - name: nginx
          image: nginx
          ports:
          - containerPort: 80                 

• Service

  apiVersion: v1
  kind: Service
  metadata:
    name: nginx-service
    labels:
      app: nginx
      svc: nginx-test
  spec:
    selector:
      app: nginx
    ports:
    - protocol: TCP
      port: 8080
      targetPort: 80

• Important commands:

• Check the IP Addresses

  kubectl get all -o wide

• Check the IP + Endpoints

  kubectl describe svc nginx

• Check Endpoints

  kubectl get ep

• Check every label

  kubectl get XXX --show-labels

• Filter based on labels

  kubectl logs/get/etc -l <label>=<value>

• Kube Proxy forwards the request
• Responsible for maintaining a list of Service IPs and corresponding Pod IPs

We can scale our application Up/Down by editing the deployment.yaml file of that deployment. But for quickly testing something:

kubectl scale deployment nginx-deployment --replicas=5

We can track our changes in cluster via --record flag...

• --record flag is almost valid for most of the K8s components (scale/create/etc)

  kubectl scale deployment nginx-deployment --replicas=5 --record

then you see it via:

1. Command line

   kubectl rollout history deployment nginx-deployment

2. in annotations

   kubectl get deployments.apps nginx-deployment -o yaml | less

• We will curl to Nginx Service from inside that Pod
• No need for a Deployment Configuration File

kubectl run test-nginx-svc --image=nginx

Then we want to Get Interactive Terminal

kubectl exec -it test-nginx-svc -- bash

Then we want to curl the nginx app via its service:

1. IP:PORT

   curl http://<SERVICE-IP>:8080

2. DNS

   curl http://nginx-service:8080

DNS Server is:

• Nameserver in the K8s cluster

• Manage the list of Service Names and their IP addresses

• All Pods point to this nameserver

• DNS Server in Kubernetes is: CoreDNS

• CoreDNS Pods run in kube-system namespace

• 2 Replicas in the kube-system namespace

• In the production cluster, minimum of 2 replicas

• Check logs of CoreDNS

  kubectl logs -n kube-system -l k8s-app=kube-dns

kubectl run -it test-nginx-svc --image=nginx -- bash

Inside that:

cat /etc/resolv.conf 

The result would look like:

nameserver 10.96.0.10 # IP address of CoreDNS
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

• kubelet automatically creates /etc/resolv.conf file for each Pod

• You can see the cluster DNS IP by: sudo cat /var/lib/kubelet/config.yaml

• <servicename>.<namespace>.svc.cluster.local
• We don't have to use the fully qualified name
• By default, K8s research only in the namespace
  • Same namespace: Only the name is sufficient
  • Different namespaces: You need to include that namespace

1. As we already know, the Pod IP Addresses come from CNI.
2. Api-server, Etcd, Kube-Proxy, Scheduler, and controller-Manager IP Addresses come from Server/Node IP Address
3. But how about the Services in K8s (ClusterIP Type)?

sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml

Our goal here is to create a service that balances connections to two different deployments. You might use this as a simplistic way to run two versions of your apps in parallel.

In the real world, you'll likely use a 3rd party load balancer to provide advanced blue/green or canary-style deployments. Still, this assignment will help further understand how service selectors are used to finding pods to use as service endpoints.

For simplicity, version 1 of our application will use the NGINX image, and version 2 will use the Apache image. They both listen on port 80 by default.

When we connect to the service, we expect to see some requests served by NGINX and some by Apache.

Let's create a clusterIP service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: custom-lb
  name: custom-lb
spec:
  type: ClusterIP
  # The selector of that service will need to match the pods created by both deployments.
  selector:
    svc: clb
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80

Then create our Nginx Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      # We will need to change the deployment specification to add an extra label (svc: clb) to be used solely by the service.
      labels:
        app: nginx
        svc: clb
    spec:
      containers:
        - image: nginx
          name: nginx
          ports:
            - containerPort: 80

Then create our Apache Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: apache
  name: apache
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apache
  template:
    metadata:
      # We will need to change the deployment specification to add an extra label (svc: clb) to be used solely by the service.
      labels:
        app: apache
        svc: clb
    spec:
      containers:
        - image: httpd
          name: httpd
          ports:
            - containerPort: 80

Apply the files, then use the curl command to show yield responses from NGINX and Apache.

Note: you won't see a perfect round-robin, i.e., NGINX/Apache/NGINX/Apache, etc., but on average, Apache and NGINX should serve approximately 50% of the requests each.

• The "pod-to-pod network" or "pod network":
  • Provides communication between pods and nodes
  • Is generally implemented with CNI plugins
• The "pod-to-service network":
  • Provides internal communication and load balancing
  • Is generally implemented with kube-proxy
• Network policies:
  • Provide firewalling and isolation
  • Can be bundled with the "pod network" or provided by another component
• Inbound traffic can be handled by multiple components:
  • Something like kube-proxy (for NodePort services) (Don't worry. We're going to go through NodePort in the next section)
  • Load balancers (ideally, connected to the pod network)
• It is possible to use multiple pod networks in parallel (with "meta-plugins" like CNI-Genie or Multus)

• ClusterIP = Internal Service
• NodePort = External Service

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  labels:
    app: nginx
    svc: nginx-test
spec:
  type: NodePort # ClusterIP is default, thats why we didnt need to specify the type before
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
      nodePort: 30000

• Internal Service accessible on Service IP address + Port
• NodePort also creates ClusterIP internal Service
• NodePort, Opens Port on each Worker Node
• External traffic has access to fixed port on each Worker Node!
• Range: 30000-32676
• Sometimes, it's the only available option for external traffic (e.g. most clusters deployed with kubeadm or on-premises)

NodePort Service Accessibility:

• NodePort accessible from outside the cluster
• On IP address of Node + The NodePort
• NodePort accessible from outside the cluster

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  labels:
    app: nginx
    svc: nginx-test
spec:
  type: LoadBalancer # The configuration of Loadbalancer is exactly like NodePort except this part...
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
      nodePort: 30000 # Notice we still have a nodePort!

How Loadbalancer Type works:

• Loadbalancer outside of K8s cluster
• Which accepts traffic as entrypoint
• Loadbalances to 1 of Worker Node on the Nodeport
• Then gets forward to ClusterIP
• Accessible at own IP address & port
• Loadbalancer is not created inside the cluster

1. Who is responsible for creating the loadbalancer?

• Cloud providers, which offer Kubernetes managed services (EKS, AKS, GKE, etc)

2. But, I have a self-managed K8s cluster

• No Loadbalancer out-of-the-box... You need to create your own Loadbalancer
• Create Loadbalancer for each Service

Some notes:

• When using managed K8s Service, like EKS:
• Cloud native loadbalancer will be provisioned for your Service
• Here we are going to create a loadbalancer from scratch on AWS

1. From EC2 managment console go to Load Balancing/Load Balancers
2. Choose Application Load Balancer
3. Give it some name (Doesn't actually matter!)
4. In Availibility Zones section, we should select a zone that our Worker Nodes are
5. Then you must select at least 2 subnets (Very specific to AWS)
6. In Configure Routing (This the part that we decide where does Loadbalancer forward the requests that it gets):

• Select a name
• In port section, we give it The nodePort from Service configuration (In our case 30000)

7. In Register Targets, we should specify Which instances this are exactly, In our case these are the Worker Nodes (Only the Worker Nodes NOT master) and then Register
8. After a while your Loadbalancer will go from Provisioning to Active state

• MetalLB website
• MetalLB Installation
• Weave & MetalLB

1. Self-Managed K8s cluster: Create Loadbalancer yourself
2. Managed K8s cluster: Creates Loadbalancers automatically

• Loadbalancer Disadvantages:

1. Loadbalancer that all become entrypoints
2. Configure Domain Name
3. Each Loadbalancer exposes new NodePort
4. Each Loadbalancer increases cloud bill
5. Configure everything outside the cluster
6. Isn't it good:

• Having this as part of K8s cluster?
• Configure secure connection
• Loadbalancing to different services

• Ingress
• A K8s component
• Configure Routing
• Configure https
• Ingress is deployed and available inside cluster
• We need to expose it either as NodePort or Loadbalancer
• 1 NodePort or Loadbalancer, which is single entrypoint

• Valid domain address
• Map domain name to the IP address, which is the entrypoint

• You need an implementation of Ingress
• which is Ingress Controller

• Evaluates and processes Ingress rules
• Manages redirection
• Entrypoint to cluster
• Many third-party implementations (Default: K8s Nginx Ingress Controller)

1. Cloud Service provider:
   • Out-of-the-box K8s solution
   • own virtualized Loadbalancer
   • Advantage: You don't have to implement Loadbalancer yourself

2. Bare Metal:
   • You need to configure some kind of entrypoint
   • Either inside of cluster or outside as separate server (Software or Hardware solution)
   • Separate server
   • Public IP address and open ports
   • Entrypoint to cluster
   • No server in K8s cluster is accessible from outside!

• Multiple paths for same host

• Multiple sub-domains or domains

• Important Notes:
  1. Data Keys need to be tls.crt and tls.key
  2. Values are file contents NOT file paths/locations
  3. Secret component must be in the same namespace as the Ingress component

We deploy our Ingress Controller via Helm charts...

• Official documentation

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install [RELEASE_NAME] ingress-nginx/ingress-nginx

Now check this out:

helm ls # Check for the chart that we just deployed
kubectl get pod # Check for the ingress controller pod
kubectl get svc # Check for 2 ingress services (ClusterIP & Loadbalancer)

Now you may wonder why a Loadbalancer Service type?

Now we should Configure a LB again... Check this section

At this point if you check the LB IP address, you should get a Nginx Error (504 Gateway Time-out OR 404 Not Found). It is because we didn't configure it yet... So as a next step: Configure Ingress Controller to route the traffic

• Ingress component is like a Configuration piece for the Ingress Controller
• We defined HTTP rules, which the Ingress Controller fulfills
• K8s way to configure routing logic
• Our routing logic

1. Change the ingress-service back to ClusterIP Service

• Only accessible internally!
• The Ingress Controller is the Only Entrypoint of the K8s cluster

First, lets generate an Ingress Template

kubectl create ingress my-app-ingress --rule=host/path=service:port --dry-run=client -o yaml > my-ingress.yaml

Open the file and cleaning up... After this, you should have something like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
spec:
  rules:
    - host: host
      http:
        paths:
          - backend:
              service:
                name: service
                port:
                  name: port
            path: /path
            pathType: Exact

Ingress Rules:

Each HTTP rule contains:

• An optional host
• List of paths, each of which has an associated backend
• A backend is combination of Service and port names

Path Types:

• Exact: Matches the URL path exactly and with case sensitivity
• Prefix: Matches based on a URL path prefix split by /

We should change this template to:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  namespace: default
spec:
  ingressClassName: nginx
  rules:
    - host: www.kube.com # The Loadbalancer Domain (MUST be domain, NOT IP address)
      http:
        paths:
          - pathType: Exact
            backend:
              service:
                name: nginx-service
                port:
                  number: 8080
            path: /my-app

• apply it

kubectl apply -f my-ingress.yaml

• Notes: If you get ...Faild calling webhook "validate.nginx.ingress.kubernetes.io...":

• See the API

kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io

• Edit the manifest file

kubectl edit validatingwebhookconfigurations.admissionregistration.k8s.io

• Change the failurePolicy

# Change the failurePolicy: Fail to the:
failurePolicy: Ignore

• Source

And now Everything should be fine:

kubectl get ingress

• Ingress component is in the same namespace as our internal Service

Now if you paste your URL in the browser, you should see the Nginx result (Instead of the errors that we got before...)

• Note: If tou are in bare-metal, you should also add the IP URL in your /etc/hosts --> e.g.

  10.152.183.110 www.kube.com
  

Now lets change our my-ingress.yaml file to:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  namespace: default
spec:
  ingressClassName: nginx
  rules:
    - host: www.kube.com
      http:
        paths:
          - pathType: Exact
            backend:
              service:
                name: nginx-service
                port:
                  number: 8080
            path: /my-app

First check the described ingress:

kubectl describe ingress my-ingress

As you can see, if now we check the URL we get an error, bit if we head over to URL/my-app we get 404 Not Found... This means now traffic will forward to the nginx-service, but my nginx doesn't know how to handle it... So, Now we want to handle this issue that our main page/root (URL) doesn't response. Just add the following line under the metadata section:

  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /

• If your app can handle the path, then you don't need this rewrite
  • Source

Now if you head over to URL/my-app, we can see the Nginx welcome page

If you check the pods, you can see that we have only 1 ingress nginx controller, In Production, you should have 1 replica per Worker Node

• Role:

  • With the Role component, you can define namespaced permissions
  • Bound to a specific Namespace
  • What resources in that namespace can you access? (Pod, Deployment, Service, ...)
  • What action can you do with this resource? ("list", "get" "update", "delete", ...)
  • Define Resources and Access Permissions
    • No information on WHO gets these permissions
  • How to attach Role Definition to a person or team?

• RoleBinding:

  • Link ("Bind") a Role to a User or Group
  • All members of the Group get permissions defined in the Role

How about K8s Admins?

• Managing Namespaces in a cluster

• Configuring cluster-wide Volumes

• ClusterRole:

  • Defines resources and permissions `cluster wide

• ClusterRoleBinding:

  • Link ("Bind") a ClusterRole to a User or Group

How do we create Users and Groups?

• Kubernetes doesn't manage Users natively
• Admins can choose from different authentication strategies
• No Kubernetes Objects exist for representing regular user accounts

1. Static Token File
2. Certificates
3. 3rd Party Identity Service

• Admin configure external source
• API Server handles authentication of all the requests
• API Server uses one of these configured authentication methods

---

• Example for Tokens (users.csv):

  aWpoZGZzaXNoaWZsa2pldWwK, user_1, u1001,group1
  aWpoZGZzaXNoaWZoc2Roc2Ru, user_2, u1002,group2
  aWpoZGZzaXNoaWZzc3Nzc3MK, user_3, u1005,group3
  

• Pass token file via --token-auth-file=/users.csv command option

  kube-apiserver --token-auth-file=/users.csv [other options]

---

Admins:

1. Manually create certificates for users
2. Or configure LDAP as the authentication source

How about Authorization for Applications

• Applications are inside the cluster and outside the cluster

• e.g.:

  1. Monitoring Apps, which collects metrics from other apps within the cluster (Prometheus)
  2. CI/CD Server deploying apps inside the cluster (Jenkins)
  3. Infrastructure Provisioning Tools configuring the cluster (Terraform)

• We also want Apps to have only the permission it needs!

• ServiceAccount: K8s Component that represent an Application as User

  • Link ServiceAccount to Role with RoleBinding (e.g., CI/CD example)
  • Link ServiceAccount to ClusterRole with ClusterRoleBinding (e.g., Monitoring Apps)

• Role Configuration File

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: my-app
  name: developer
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
      - list
      - watch
    resourceNames: # Optional
      - "myapp"
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
    resourceNames: # Optional
      - "myapp"

• apigroups: "" Indicates the core API group
• resources: K8s components like Pods, Deployments, etc.
• verbs:
  • The actions on a resource
  • get, list (read-only) or create, update (read-write)
• resourceNames: Define access to only certain pods in that namespace

• RoleBinding Configuration File

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jane-developer-binding
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: jane
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: developer

• ClusterRole Configuration File

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-admin
rules:
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
      - list
      - create
      - delete
      - update

1. Define access for cluster-wide resources
2. Define access for namespace resources

• Create Role, ClusterRole, etc. just like any other Kubernetes component

  kubectl apply -f <ROLE-MANIFEST.YAML>

• View Components with get and describe command

  kubectl get roles
  kubectl describe role <NAME>

• Kubectl provides an auth can-i subcommand
• To quickly check if the current user can perform a given action

kubectl auth can-i create deployments --namespace dev

• Admins can also check the permissions of other users

Layers of Security

Let's say our Jenkins application from outside the cluster wants to connect to the cluster:

1. API Server checks if Jenkins User is authenticated
2. Is Jenkins allowed to connect to the cluster at all?
3. You can enable multi-authentication methods at once
4. RBAC is one of the multiple Authorization Modes
5. With RBAC: Role, ClusterRole, and Bindings are checked

RBAC is one of four authorization modes...

• 4 Authorization Modes:

  1. Node
  2. ABAC
  3. RBAC
  4. Webhook

• Kubernetes Docs

• You can choose more than one authorization modules

Where to enable the authorization mode?

• In the API Server configuration

  sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml

and check the - --authorization-mode=Node,RBAC under the command section:

spec:
  containers:
    - command:
        - kube-apiserver
        - --advertise-address=192.168.220.121
        - --allow-privileged=true # Enabled by default
        - --authorization-mode=Node,RBAC

We already saw that when we initialized our cluster, kubeadm generated certificates automatically and stored them:

• Server Certificate in /etc/kubernetes/pki (apiserver.crt & apiserver.key)
• ETCD Certificate in /etc/kubernetes/pki/etcd

Kubeadm inits also generated a Client Certificate for other services that talk to the API Server.

How signed those Certificates?

• Kubeadm generated a CA for K8s cluster
• All clients have a copy of k8s CA
• K8s CA's are trusted within K8s by all components, who have a copy of the CA

As we said before, K8s allow you to configure these external sources;

But you have to manage them yourself in this section, we talk about Configure User Authentication via Certificates.

In this overview:

• User Account
  • Create a client key with openssl
  • Create CertificateSigningRequest for key
  • Approve CertificateSigningRequest
  • Get signed certificate
  • Permit to create, delete, and update K8s resources
  • Validate user permission
• Service Account
  • Create a Service Account for Jenkins
  • Give Permissions to create, delete, and update K8s resources

kubectl certificate approve dev-tom

kubectl get csr dev-tom -o yaml

echo 'BASE64 CODED CERTIFICATE' | base64 --decode > dev-tom.crt

kubectl \
--server https://192.168.220.121:6443 \ # The address from above command
--certificate-authority /etc/kubernetes/pki/ca.crt \
--client-certificate dev-tom.crt \
--client-key dev-tom.key \
get pod

# [ ... ]
users:
  - name: dev-tom
    user:
      client-certificate: dev-tom.crt
      client-key: dev-tom.key

• Is a Pod running?

  kubectl get POD_NAME

• Is Pod registered with Service?

• Is Service forwarding the request?

  kubectl get ep

  kubectl describe SERVICE_NAME

• Is Service accessible?

  nc SERVICE_IP SERVICE_PORT

  ping SERVICE_NAME

• Check application logs

  kubectl logs POD_NAME

• Check Pod status and recent events

  kubectl describe pod POD_NAME

These were just Simple paths of troubleshooting Pod and Application

• In previous sections, we learned Debugging inside the Pod
• Now, we are going to Debug cluster components using kubectl
• Exact information in a digestible way

• Get a massive list of attributes about our nodes:

  kubectl get no -o json

But this is too much! well, Kubectl uses JSONPath expressions to filter on specific fields in the JSON object and format the output...

• -o wide: Standard output with additional information.
• -o yaml: Output a YAML formatted object.
• -o json: Output a JSON formatted object.

What if one of our worker nodes gets into NotReady status?! This problem is most commonly for kubelet, So, Check the Kubelet process.

• Check kubelet status:

  service kubelet status

• Check extended logs of Kubelet service with journalctl:

  journalctl -u kubelet

• Check where the kubelet is:

  which kubelet

• Open the kubelet configuration (Check the path via service kubelet status):

  sudo vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

  # Note: This dropin only works with kubeadm and kubelet v1.11+
  [Service]
  Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
  Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
  # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
  EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
  # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
  # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
  EnvironmentFile=-/etc/default/kubelet
  ExecStart=
  ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

• in the ExecStart section, check the /usr/bin/kubelet path is the same path as you got from which kubelet

• Reload the service

  sudo systemctl daemon-reload
  sudo systemctl restart kubelet

• Check the kubelet service again

  service kubelet status

If you can not connect to the cluster: Check the ~/.kube/config file:

• Check if CA Certificate is correct?

  echo 'CA-CERTIFICATE' | base64 --decode

  sudo cat/etc/kubernetes/pki/ca.crt

  • This two value should be the same.
• Check if server endpoint is correct?

• You can have multiple containers inside the Pod
• Main and Helper application
• The container providing helper functionality is called the sidecar container
• Usually operates asynchronously
• Can talk to each other using localhost (WithOUT DNS, IP, etc.)
• Can share data

• e.g.:
  • Set Environment Variables
  • System Checks
  • Wait for service to be available
• Run once in the beginning and exits
• the Main container starts afterward
• Init containers are used to initialize something inside your Pod

  apiVersion: v1
  kind: Pod
  metadata:
    name: myapp-pod
  spec:
    containers:
      - name: myapp-container
        image: busybox:1.8
        command: ['sh', '-c', 'echo the app is running!']
    initContainers:
      - name: init-myservice
        image: busybox:1.28
        command: ['sh', '-c', 'COMMAND']

• Create a file and apply it

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: nginx-deployment
    labels:
      app: nginx
  spec:
    replicas: 3
    selector:
      matchLabels:
        app: nginx
    template:
      metadata:
        labels:
          app: nginx
      spec:
        containers:
        - name: nginx
          image: nginx
          ports:
          - containerPort: 80
        - name: log-sider
          image: busybox
          command: ['sh', '-c', "while true; do echo sync app logs; sleep 20; done"]                                                                                                         

• Checks these things out:
  • Pods and their creation
  • the pod logs (when you have multiple containers, you always need to specify the container!)

• Adding an Init container to the previous file:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: nginx-deployment
    labels:
      app: nginx
  spec:
    replicas: 3
    selector:
      matchLabels:
        app: nginx
    template:
      metadata:
        labels:
          app: nginx
      spec:
        containers:
        - name: nginx
          image: nginx
          ports:
          - containerPort: 80
        - name: log-sider
          image: busybox
          command: ['sh', '-c', "while true; do echo sync app logs; sleep 20; done"]
        initContainers:
          - name: mydb-available
            image: busybox
            command: ['sh', '-c', "until nslookup mydb-service; do echo waiting for database; sleep 4; done"]

  • Check that the pods remain at the Init state...
• Create a service and see that the Pods state will change!

  kubectl create svc clusterip mydb-service --tcp=80:80

Let's say you need some data about your application's Pod or K8s environment to add Pod information as metadata to logs. Such as e.g.

• Pod IP

• Pod Namespace

• Service Account of Pod

• But how to access this information?

  • All Pod information can be made available in the config file

• There are two ways to expose Pod fields to a running Container:

  1. Environment Variables
  2. Volume Files

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-env
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
        - name: log-sider
          image: busybox
          command: [ 'sh', '-c' ]
          args:
            - while true; do
              echo sync app logs;
              printenv POD_NAME POD_IP POD_SERVICE_ASCCOUNT;
              sleep 20;
              done;
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: POD_SERVICE_ASCCOUNT
              valueFrom:
                fieldRef:
                  fieldPath: spec.serviceAccountName

Introduction to Kubernetes Volumes, How to persist data in K8s using volumes?

• Persistent Volume
• Persistent Volume Claim
• Storage Class

Where is the data persisted?

1. Remote Storage Volumes
   • Cloud-storage
   • Remote-storage
2. local Volumes
   • Data is persisted on the K8s Node

• You should use remote storage, especially in production!
• Remote storage systems persist data independent of the K8s Node, where the data originated.

• hostPath Volume
  • Simple configuration
  • But use only for single node testing. For multi-node clusters: use the local volume type instead.
  • Also, hostPath volumes contain many security risks, and it's best practice to avoid the use of hostPaths when possible
• NOTE: This is one of the volume types tested in the CKA Exam!

Lecture Overview

1. Create PV
   • Data is stored on the local machine
2. Create PVC
3. Create a Deployment to use Volume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-data
spec:
  hostPath: # Type of PV
    path: "/mnt/data" # Depending on the type, the attributes differ
  capacity: # Capacity
    storage: 10Gi # A PV will have a specific storage capacity
  accessModes:
    - ReadWriteOnce

kubectl get pv

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-data-pvc
spec:
  resources:
    requests:
      storage: 5Gi # the specification doesn't need to match 100%, but it must be most fitting option
  accessModes:
    - ReadWriteOnce

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-db
  labels:
    app: my-db
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-db
  template:
    metadata:
      labels:
        app: my-db
    spec:
      containers:
        - name: mysql
          image: mysql:8.0
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: mypwd
          volumeMounts:
            - name: db-data
              mountPath: "/var/lib/mysql"
      volumes:
        - name: db-data
          persistentVolumeClaim:
            claimName: mysql-data-pvc

• Logging Use Case
  • Can't access each other's file system
  • Doesn't need to be persisted
• Caching Use Case
  • Storage can be shared
  • No persistence necessary

emptyDir Volume

• Suitable for multi-container Pods
• All containers in the Pod can read and write the same files in the emptyDir Volume
• EmptyDir volume is initially empty
• Is first created when a Pod is assigned to a Node and exists as long as that Pod is running on the Node
• When Pod is removed from a Node, the Data is deleted permanently
• Data inside the emptyDir is mounted into the containers file system

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-vol
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
          command: [ 'sh', '-c' ]
          args:
            - while true; do
              echo "$(date) INFO some app date" >> /var/log/myapp.log;
              sleep 5;
              done;
          volumeMounts:
            - name: log-data
              mountPath: /var/log
        - name: log-sider
          image: busybox
          command: [ 'sh', '-c' ]
          args:
            - tail -f /var/sidecar/myapp.log;
          volumeMounts:
            - name: log-data
              mountPath: /var/sidecar
      volumes:
        - name: log-data
          emptyDir: { }

kubectl logs pod/nginx-deployment-vol-7b7bf97fdf-gl96t -c log-sider

Steps:

1. Create ConfigMap component
2. Create Secret component
3. Pass Data to Pod using Evironment Variables

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config
data:
  db_host: mysql_service

apiVersion: v1
kind: Secret
metadata:
  name: myapp-secret
type: Opaque # Have many type
data: # Values must be "base64" encoded stings.
  username: bXl1c2Vy # See below command
  password: bXlwd2Q=

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  replicas: 1
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: busybox
          command: [ 'sh', 'c', "printenv MYSQL_USER MYSQL_PWD MYSQL_SERVER; sleep 10" ]
          env:
            - name: MYSQL_USER
              valueFrom:
                secretKeyRef:
                  name: myapp-secret
                  key: username
            - name: MYSQL_PWD
              valueFrom:
                secretKeyRef:
                  name: myapp-secret
                  key: password
            - name: MYSQL_SERVER
              valueFrom:
                configMapKeyRef:
                  name: myapp-config
                  key: db_host

Often applications use a configuration File, rather than just individual values

1. Create configuration file via ConfifMap & Secret
2. Pass to Pod configuration via Volume

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config-file
data:
  mysql.conf: |
    [mysqld]
    port=3306
    socket=/tmp/mysql.sock
    key_buffer_size=16M
    max_allowed_packet=128M

apiVersion: v1
kind: Secret
metadata:
  name: myapp-secret-file
type: Opaque
data:
  secret.file: |
    "base64" ENCODED FILE VALUE VIA --> base64 FILE | tr -d "\n"

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-db
  labels:
    app: my-db
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-db
  template:
    metadata:
      labels:
        app: my-db
    spec:
      containers:
        - name: mysql
          image: busybox:1.28
          command: [ 'sh', '-c', "cat /mysql/db-config/mysql.conf; /mysql/db-secret/secret.file; sleep 20" ]
          volumeMounts:
            - name: db-coinfig
              mountPath: /mysql/db-config
            - name: db-secret
              mountPath: /mysql/db-secret
              readOnly: true
      volumes:
        - name: db-coinfig
          configMap:
            name: myapp-config-file
        - name: db-secret
          secret:
            secretName: myapp-secret-file

• When updating configMap or secret, Pods don't get the up-to-date data automatically.
• for that, you have to restart the pods.

  kubectl rollout restart deployment/my-db

• 2 Types of resources:

  1. CPU
  2. Memory

• Configure resource requests for each container.

  • Requests is what the container is grantedd to get
  • K8s Scheduler uses this to figure out where to run the Pods

Why resource limits are so important?

• Container will consume more than the requested resources.
• If not limited,container could consume all the Node's resources.

Configure resource requests for each container.

• Make sure container never goes above a certain value.
• Container is only allowed to go up to the limit.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-resource
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: my-app
          image: nginx
          resources:
            requests:
              memory: "64Mi" # Memory resources are defined in "bytes"
              cpu: "250m" # CPU resources are defined in "millicores"
            limits:
              memory: "128Mi"
              cpu: "500m"
        - name: log-sider
          image: busybox
          command: [ 'sh', '-c', "while true; do echo sync app logs; sleep 20; done" ]
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "200m"

• Best practice: Keep CPU and request at 1 or below
• If you put values larger than your biggest Node, your Pod will never be scheduled!

kubectl get pod -o jsonpath="{range .items[*]} {.metadata.name}{.spec.containers[*].resources}{'\n'}"

• nodeSelector
  • More flexibility over nodename
  • If not enough resources, Pods won't be scheduled
  • More flexible expressions

apiVersion: v1
kind: Pod
metadata:
  name: with-node-affinity
spec:
  containers:
    - image: nginx
      name: myapp
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                  - linux
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
              - key: type
                operator: In
                values:
                  - cpu

• Properties of Pods
  • nodeName, nodeSelector, nodeAffinity
• Configure Nodes... Which Pods they should accept to be scheduled on?
• e.g. Worker1 & Worker2 should repel my-app Pods
• Master Node also have Taints

• See the Taints attribute

  kubectl describe node master

• See that the Taints is set only on Master Nodes

  kubectl describe node | grep Taints

• Who set the taints on Master Nodes?
  • When bootstrapping kubeadm sets the taint

But we can see there are a bunch of Pods there! How did the Pods land on the Control Plane?

• Taints
  • Applied to Nodes
• Toleration
  • Applied to Pods
  • Allow the pods to schedule onto Nodes with matching taints
  • From No Tolrations to Has a matching toleration
• So, Taints and Tolerations work together

• Check the existing Pods Toleration on master nodes

  kubectl describe pod -n kube-system etcd-master

• They all return Tolerations: :NoExecute op=Exists

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-toleration
spec:
  containers:
    - image: nginx
      name: nginx
  tolerations:
    - effect: NoExecute
      operator: Exists
  nodeName: master # Why we add this? Check the below:

Let's say we have a Dynamic infrastructure that Nodes get added and removed based on workload. And we have a log-collector the collecting logs on Master nodes on 5 Nodes (5 Master Nodes). If we specify nodeSelector to be scheduled on Master Nodes, All replicas might be scheduled on that 1 Control Plane Node! But we wanted: 1 replica per Node...

• Allows you to constrain, which Nodes your Pod is eligible to be scheduled, based on labels on Pods that are already running on the Node
• In other words: This Pod should not run on worker1 if worker1 is already running one or more Pods with a specific label.
• In our example: Don't run Pod on a Node that already runs this Pod's replica

• Imagine: Our app works with etcd only
• We only want to Let's say we have a Dynamic infrastructure the app on those Nodes that run etcd

• Inter-Pod Affinity
  • Schedule our Pods only on Nodes that have etcd running on them
• Inter-Pod Anti-Affinity
  • And which do not have another replica of my-app already running

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: myapp
  name: myapp-deployment
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - image: nginx
          name: myapp-container
      tolerations:
        - effect: "NoSchedule" # k describe node master | grep Taint
          operator: "Exists"
      nodeSelector:
        type: master
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - "myapp"
              topologyKey: "kubernetes.io/hostname"
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: tier
                    operator: In
                    values:
                      - "control-plane"
              topologyKey: "kubernetes.io/hostname"

• Deployment
  • Creates an application rollout
• ReplicaSet
  • Created automatically in the background
  • Ensure that a specified number of Pod replicas are running at any qiven time

 ________________            ________________            __________
/   Deployment   \ ------>  /   ReplicaSet   \ ------>  /   Pods   \ 
 -----------------          ------------------          ------------

• We work with Deployment
• K8s creates ReplicaSet in background
• ReplicaSet creates Pods

• In which order do Pods get removed and new ones created?
  1. Recreate Strategy: All existing Pods are killed before new ones are created (But it cause the Application Downtime ❌)
  2. Rolling Update Strategy: The Deployment updates Pods in a rolling update fashion (No Application Downtime ✅)
• ReplicaSet and its Pods are linked.
• Name of ReplicaSet: [Deployment-Name]-[Random-string]

If you describe one of your existing Deployments. you should see:

$ kubectl describe deployment nginx-deployment

[...]
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
[...]

• You can specify how many Pods to update at once
  • Instead of deleting/creating Pods one by one
  • You may want to update 5 at once
• Max Unavailable: Specifies the max number of Pods that can be unavailable during the update process
• Max Surge: Specifies the max number of Pods that can be created over the desired number of Pods

$ kubectl rollout history deployment nginx-deployment

deployment.apps/nginx-deployment 
REVISION  CHANGE-CAUSE # Here we have 5 revision
1         <none>
2         <none>
4         <none>
5         <none>

• Owners and Dependents
• Some objects are created by other objects (example: Pods created by replicaSets, themselves created by Deployments)
• When an owner object is deleted, its dependents are deleted (this is the default behavior; it can be changed)
• We can delete a dependent directly if we want (but generally, the owner will recreate another right away)
• An object can have multiple owners

• When deleting an object through the API, three policies are available (Docs):
  • foreground (API call returns after all dependents are deleted)
  • background (API call returns immediately; dependents are scheduled for deletion)
  • orphan (the dependents are not deleted)
• When deleting an object with kubectl, this is selected with --cascade:
  • --cascade=background deletes all dependent objects (default)
  • --cascade=orphan orphans dependent objects

We are going to delete the Deployment and replicaSet that we created without deleting the corresponding Pods!

• Delete the Deployment

  kubectl delete deployment owner-example --cascade=orphan

• Delete the replicaSet

  kubectl delete rs -l app=owner-example --cascade=orphan

• Check that the pods are still here

  kubectl get pods

• etcd: A distributed, reliable key-value store
• Periodically backing up the etcd cluster data is important

• What is stored inside etcd?
  • Kubernetes cluster configuration and state data such as the number of pods, their state, namespace, etc. It also stores Kubernetes API objects and service discovery details.
• What is not stored inside etcd?
  • Application Data is not stored in it
  • Reminder: Storage configured with Persistent Volume for application data

• How to create an etcd Backup?
  • Etcdctl is a command line tool for interacting with etcd server

1. Install etcdctl
2. Create backup with etcdctl

ETCDCTL_API=3 \ # The API version used by etcdctl to speak to etcd, you can check the version with "etcdctl version"
etcdctl snapshot save /tmp/etcd-backup-new.db

• How about:

  • Multiple replicas?
  • Manage etcd independent of K8s cluster?

• See the location of etcd

  $ sudo cat /etc/kubernetes/manifests/etcd.yaml
  
    [ ... ]
    - hostPath:
        path: /var/lib/etcd
        type: DirectoryOrCreate
      name: etcd-data

1. Use Remote Storage outside K8s cluster
   • AWS, Google Cloud, etc.
   • On-Permise Storage
2. Run etcd outside K8s cluster
   • Instead of running etcd on Master Nodes, run them outside cluster
   • More complex, but options you should consider

We need two things to connect to API:

1. location of cluster (= Endpoint of API Server)
2. Credentials to authenticate

• First way: access using kubectl
  • with kubeconfig --> Till now we were used this method (Default method)
• Second way: Directly accessing the REST API
  • with curl, wget, browser

We will discuss about the Second way in this chapter

• Kube proxy:

  • Kubectl will run in proxy mode
  • It uses the stored apiserver location and verifies the identity of the API server using a self-signed cert

• Start the proxy

  kubectl proxy --port 8080 &

• check the API Server

  curl http://localhost:8080/api/v1

• Another way is without kubectl proxy
  • We need to pass authentication ourselves
• In this lecture, we execute script with a user with limit permissions

How Cluster Upgrade works?

1. Upgrade Cintrol Plane
   • Cons
     • Control Plane processes not available
     • No application downtime
     • Mangement functionalities not available
     • Crashed Pods won't be rescheduled
   • Solution
     • Have more than 1 Control Plane Node
     • Upgrade each Node one by one
2. Upgrade Worker Nodes

In this chapter, we will go through updating the K8s cluster with all the best practices...

Be sure to check the Upgrading kubeadm clusters docs

Lets say we have multiple clusters to administrator, so we have multiple kubeconfig file.

But it's not so efficent to use --kubeconfig option everytime with our kubectl command!

Access multiple clusters using Contexts

• Define all the clusters and users in the 1 kubeconfig file
• Define a context for each cluster
• We can switch between clusters using these contexts
• No need to specify kube konfig file

___________________________     _____________________________    ______________________________ 
|   stage-admin@staging    |    |   dev-admin@development    |   |   my-script@development    |
| stage-admin ---> staging |    | dev-admin ---> development |   | my-script ---> development |
|__________________________|    |____________________________|   |____________________________|

contexts:
  - context:
      cluster: development
      user: my-script
    name: my-script@development

We already know that all certificates has been generated by Kubeadm...

• Check the certs expirations

  sudo kubeadm certs check-expiration

• Client certificates expire after 1 year

• CA certificates expire after 10 year

• Check certs with detail

  sudo openssl x509 -in /etc/kubernetes/pki/ca.crt -text -noout 

• Check certs with detail and grep the expiration

  sudo openssl x509 -in /etc/kubernetes/pki/ca.crt -text -noout | grep Validity -A2