October, 2020

Liveness and Readiness probes are required in Kubernetes to prevent deadlock of your application deployed and zero missing request while pod is initializing. When probe is configured in deployment, each pod will go through probe conditions.

Liveness and readiness probes will be applicable to new pod which is created by horizontal pod autoscaling (hpa)

We are going to learn how to configure probes in kubernetes deployment:

Liveness probe: It will take care of the container when it is in deadlock or application not running by restarting container
Readiness probe: It will take care of the pod when to join the service to serve traffic

Prerequisite:

1) kubectl cli should be installed in your local
2) Connect to the kubernetes cluster
3) /health API should be enabled in application
4) Configmaps related to deployment should be deployed before deployment

Step1:

Configure the liveness and readiness probe in the deployment yaml

Liveness probe:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 3
  periodSeconds: 3

Liveness probe should be configured under template.spec.containers section

Readiness probe:

readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 3
  periodSeconds: 3

Readiness probe configuration is similar to liveness probe and should be configured under template.spec.containers section

Sample Deployment Yaml with probes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: Myapp-deployment
  labels:
    apptype: java
spec:
  replicas: 3
  selector:
    matchLabels:
      apptype: java
      tier: web
  template:
    metadata:
      labels:
        apptype: java
        tier: web
    spec:
      containers:
      - name: [container name]
        image: [image from any container registry]
        command: ["/bin/sh"]
        args: ["java -Xmx2g -Xss256k -XX:MaxMetaspaceSize=256m -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/my-heap-dump.hprof -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/myAppgc.log -jar [application.jar] --spring.config.location=[absolute path of the config maps mounted] "]
        ports: 
        - containerPort: 8080
        volumeMounts:
        - name: appconfig
          mountPath: "/config/application"
          readOnly: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 3
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 3
      volumes:
      - name: appconfig
        configMap:
        name: [configmap-name]

Note: InitialDelaySeconds need to be higher value than application start up time

Here are the few fields we need to consider while configuring probes:

kubectl apply -f [deployment.yaml]

High availability (HA) is the ability to make our application continuously operational for a desirably long length of time. By configuring Pod Anti-affinity and Pod disruption budget together will make stateful or stateless application pods to highly available during any of the below scenarios:

1) Any one node is unavailable or under maintenance
2) cluster administrator deletes kubernetes node by mistake
3) cloud provider or hypervisor failure makes kubernetes node disappear
4) Cluster administrator/User deletes your application pods by mistake

In this blog, We are going to configure pod anti-affinity and Pod disruption budget for kubernetes deployment

Prerequisite:

Pod Anti-affinity:

Pod Anti-affinity: It help us not to schedule same type of pod in one node which means you will have only one pod in one node and other same type pod in another node. Hence scheduler will never co-locate the same type of pod in the same node

For Example, if three pods of same deployment(replicas=3) are in one node and that node has crashed or unavailable, then our application will be impacted. When pod anti-affinity is configured, one node failure/unavailable will not impact the whole application.

1) Configure the below snippet under template.spec in the application deployment yaml

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - myapp
       topologyKey: "kubernetes.io/hostname"

Note: Above Pod Anti-affinity snippet uses Selector to match with key app in value myapp and uses node labels

2) Sample deployment yaml with pod anti-affinity:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  labels:
    apptype: java
    app: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      apptype: java
  template:
    metadata:
      labels:
        apptype: java
        app: myapp
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - myapp
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: [container-name]
        image: [image from any container registry]
        command: ["/bin/sh"]
        args: ["java -Xmx2g -Xss256k -XX:MaxMetaspaceSize=256m -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/my-heap-dump.hprof -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/myAppgc.log -jar [application.jar] --spring.config.location=[absolute path of the config map eg. /config/application/[application.properties]] "]
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: appconfig
          mountPath: "/config/application"
          readOnly: true
      volumes:
      - name: appconfig
        configMap:
        name: [configmap-name]

3) Save the above mentioned deployment yaml and deploy it in kubernetes cluster by executing the below command

kubectl apply -f [deployment.yaml]

Pod Disruption Budget:

Pod Disruption Budget help us to define how many minimum available and maximum unavailable pods must meet the condition even during disruption for stateful or stateless application. You can define minAvailable/maxUnavailable values either as integer or percentage.

1) You can configure Pod disruption budget in two ways:

minAvailable – Specified value/percentage of pod should be available always

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: pdb-myapp
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: myapp

maxUnavailable – Specified value/percentage of pod can be acceptable during disruption

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: pdb2-myapp
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: myapp

Note: minAvailable and maxUnavailable cannot be both set in a single yaml. You need to create two pod disruption budget if you are configuring both minAvailable and maxUnavailable for your application deployment

2) We have already deployed our application with label app: myapp. To configure pod disruption budget, save the above two PodDisruptionBudget yamls and deploy it in the kubernetes cluster by executing the below commands:

kubectl apply -f pdbmin.yaml
kubectl apply -f pdbmax.yaml

Now Pod Disruption Budget is applied for the label app: myapp

3) Verify the pod disruption budgets by executing the below command:

kubectl get poddisruptionbudgets