Kubernetes: Why won't that deployment roll?

Kubernetes Deployments provide a declarative way of managing replica sets and pods.  A deployment specifies how many pods to run as part of a replica set, where to place pods, how to scale them and how to manage their availability.

Deployments are also used to perform rolling updates to a service. They can support a number of different update strategies such as blue/green and canary deployments.

The examples provided in the Kubernetes documentation explain how to trigger a rolling update by changing a docker image.  For example, suppose you edit the Docker image tag by using the kubectl edit deployment/my-app command, changing the image tag from acme/my-app:v0.2.3 to acme/my-app:v0.3.0.

When Kubernetes sees that the image tag has changed, a rolling update is triggered according to the declared strategy. This results in the pods with the 0.2.3 image being taken down and replaced with pods running the 0.3.0 image.  This is done in a rolling fashion so that the service remains available during the transition.

If your application is packaged up as Helm chart, the helm upgrade command can be used to trigger a rollout:

helm upgrade my-release my-chart

Under the covers, Helm is just applying any updates in your chart to the deployment and sending them to Kubernetes. You can achieve the same thing yourself by applying updates using kubectl.

Now let's suppose that you want to trigger a rolling update even if the docker image has not changed. A likely scenario is that you want to apply an update to your application's configuration.

As an example, the ForgeRock Identity Gateway (IG) Helm chart pulls its configuration from a git repository. If the configuration changes in git, we'd like to roll out a new deployment.

My first attempt at this was to perform a helm upgrade on the release, updating the ConfigMap in the chart with the new git branch for the release. Our Helm chart uses the ConfigMap to set an environment variable to the git branch (or commit) that we want to checkout:

kind: ConfigMap
data:
   GIT_CHECKOUT_BRANCH: test

After editing the ConfigMap, and doing a helm upgrade, nothing terribly exciting happened.  My deployment did not "roll" as expected with the new git configuration.

As it turns out, Kubernetes needs to see a change in the pod's spec.template before triggers a new deployment. Changing the image tag is one way to do that,  but any change to the template will work. As I discovered, changes to a ConfigMap *do not* trigger deployment updates. 

The solution here is to move the git branch variable out of the ConfigMap and into to the pod's spec.template in the deployment object:

 spec:

      initContainers:

      - name: git-init

        env:

        - name: GIT_CHECKOUT_BRANCH

          value: "{{ .Values.global.git.branch }}"

When the IG helm chart is updated (we supply a new value to the template variable above), the template's spec changes, and Kubernetes will roll out the new deployment.

Here is what it looks like when we put it all together (you can check out the full IG chart here).

# install IG using helm. The git branch defaults to "master"
$ helm install openig
NAME:   hissing-yak

$ kubectl get deployment
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE

openig    1         1         1            1           4m


# Take a look at our deployment history. So far, there is only one revision:
$ kubectl rollout history deployment
deployments "openig"
REVISION CHANGE-CAUSE
1
# Let's deploy a different configuration. We can use a commit hash here instead
# of a branch name:
$ helm upgrade --set global.git.branch=d0562faec4b1621ad53b852aa5ee61f1072575cc \ 
    hissing-yak openig

# Have a look at our deployment. We now see a new revision
$ kubectl rollout history deploy
deployments "openig"
REVISION  CHANGE-CAUSE
1         
2         
# Look at the deployment history. You should see an event where the original
# replicaset is scaled down and a new set is scaled up:
kubectl describe deployment
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
---> Time=19M ago. This is the original RS
  Normal  ScalingReplicaSet  19m   deployment-controller  Scaled up replica set openig-6f6575cdfd to 1
------> Time=3m ago. The new RS being brought up
  Normal  ScalingReplicaSet  3m    deployment-controller  Scaled up replica set openig-7b5788659c to 1
-----> Time=3m ago. The old RS being scaled down
Normal  ScalingReplicaSet  3m    deployment-controller  Scaled down replica set openig-6f6575cdfd to 0

One of the neat things we can do with deployments is roll back to a previous revision. For example, if we have an error in our configuration, and want to restore the previous release:

$ kubectl rollout undo deployment/openig
deployment "openig" rolled back
$ kubectl describe  deployment:
Normal  DeploymentRollback  1m  deployment-controller  Rolled back deployment "openig" to revision 1

# We can see the old pod is being terminated and the new one has started:
$ kubectl get pod
NAME                      READY     STATUS        RESTARTS   AGE
openig-6f6575cdfd-tvmgj   1/1       Running       0          28s
openig-7b5788659c-h7kcb   0/1       Terminating   0          8m

And that my friends, is how we roll.