3.3 Deploying as a canary deployment strategy for DevOps services

Use the Code Repository, Build Pipeline, Trigger, etc. created in 2.2 Deploying with Blue/Green Deployment Strategy of DevOps Service.

In this case, we will create a new Deployment Pipeline with only the deployment strategy where changes occur.

And change the Deployment Pipeline called in the last step of the Build Pipeline from the old Blue/Green to the Canary Deployment Pipeline.

Create a Canary Deployment Pipeline

It is possible to configure the part corresponding to the CD process of distributing the products built during CI/CD to the actual server through the Deployment Pipeline.

Deploying to Kubernetes with the canary strategy requires two things: a deployment environment, a Kubernets Manifest file to deploy, and a Kubernetes Namespace to be used for deployment.

Registering the Kubernetes Environment

Use the same environment as the Blue/Green deployment lab.

Prepare the manifest file for deployment to Kubernetes

Use the same files as the Blue/Green deployment exercise.

Create a namespace to deploy to Kubernetes

We need two namespaces where the test version and the live service will be deployed.

1. Connect to the environment where the kubectl command can be executed through Cloud Shell.

2. Create two namespaces.

   • Ex) ns-canary-stage, ns-canary-prod

     kubectl create ns ns-canary-stage
     kubectl create ns ns-canary-prod
     

   • Pre-delete the old namespace for Blue/Green for smooth testing.

     kubectl delete ns ns-blue
     kubectl delete ns ns-green
     

3. Import images from OCIR

   If OCIR repository land (eg, webpage) does not exist in advance, it is created in a private form in the Root Compartment. To import from OKE, either create an imagepullsecret in each namespace in advance, or create a public repository in advance.

Create a Kubernetes deployment stage with canary strategy

1. Go to Projects page and go to Deployment Pipelines in the left menu.

2. Click Create pipeline to create the pipeline.

   • Name: Ex) webpage-canary-deployment-pipeline

3. Click the created pipeline to add the Canary Strategy Stage.

   

4. Select the deployment type as OKE, and select the deployment environment and manifest file.

5. Make additional settings for canary deployment.

   • Canary namespace: Enter the namespace name where the test version will be deployed.

     • Ex) ns-canary-stage

   • NGINX ingress namespace: Enter the ingress resource name of the deployment app defined in the deployment manifest file.

   

6. Shift traffic: Add a stage for traffic diversion.

   • Name: Ex) shift-traffic-stage
   • Ramp Limit: Enter the percentage of the total request to be delivered to the test version. Ex) 25 percent

   

7. Approval: Automatically proceed up to deployment, and add to divert traffic after approval. You can think of it as a traffic changeover switch.

   • Name: Ex) approval-stage

   

8. Production canary

   1. Name: Ex) switch-to-production-stage
   2. Production namespace: Enter the namespace name where the actual service version will be deployed.
      • Ex) ns-canary-prod

   

9. Once the setup is complete, the following pipeline is complete.

   

Calling Deployment Pipeline from Build Pipeline

After the Build Pipeline we created earlier is finished, we add a call to the Deployment Pipeline so that it can be deployed.

1. Go to the Build Pipelines you created earlier.

2. At the end of the pipeline, right-click the three dots on the Trigger Deployment Stage > select View details.

3. Choose Edit Stage to edit.

4. Click Select deployment pipeline to save the changes to the newly created Deployment Pipeline for Canary deployment.

5. The whole flow of post-build deployment is complete.

Testing - first app deployment

1. Random changes are made to the oci-devops-oke-webpage source code specified in Trigger and reflected in the Code Repository.

2. Go to Cloud Shell with the code you are working on.

3. You can change html/index.html, but for convenience, change the Dockerfile set as an environment variable.

4. Changes are made based on the current setting, and the code is committed and then pushed.

   ...
   ENV VERSION="2.0"
   ENV MESSAGE="Hello OCI DevOps Canary"
   ENV BACKGROUND="blue"
   ...
   

5. It becomes a Trigger and the build starts.

6. The build succeeds, the deployment pipeline runs, and waits in approval-stage.

   

7. Check the Kubernetes deployment status. You can see that it is deployed only to ns-canary-stage.

   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get all,ingress -n ns-canary-stage
   NAME                                     READY   STATUS    RESTARTS   AGE
   pod/webpage-deployment-9bff68bcf-7bjk2   1/1     Running   0          4m24s
   pod/webpage-deployment-9bff68bcf-d8pwq   1/1     Running   0          4m24s
   pod/webpage-deployment-9bff68bcf-v76nt   1/1     Running   0          4m24s
   
   NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
   service/webpage-service   ClusterIP   10.96.35.238   <none>        8080/TCP   4m24s
   
   NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/webpage-deployment   3/3     3            3           4m24s
   
   NAME                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/webpage-deployment-9bff68bcf   3         3         3       4m24s
   
   NAME                                        CLASS    HOSTS   ADDRESS          PORTS   AGE
   ingress.networking.k8s.io/webpage-ingress   <none>   *       146.56.118.147   80      4m23s
   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get all,ingress -n ns-canary-prod
   No resources found in ns-canary-prod namespace.
   

8. Return to the running deployment pipeline and approve in approval-stage.

9. If approved, it will also be deployed to the Procution namespace, ns-canary-prod.

   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get all,ingress -n ns-canary-stage
   NAME                                     READY   STATUS    RESTARTS   AGE
   pod/webpage-deployment-9bff68bcf-7bjk2   1/1     Running   0          8m2s
   pod/webpage-deployment-9bff68bcf-d8pwq   1/1     Running   0          8m2s
   pod/webpage-deployment-9bff68bcf-v76nt   1/1     Running   0          8m2s
   
   ...
   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get all,ingress -n ns-canary-prod
   NAME                                     READY   STATUS    RESTARTS   AGE
   pod/webpage-deployment-9bff68bcf-4v962   1/1     Running   0          2m14s
   pod/webpage-deployment-9bff68bcf-9w7d5   1/1     Running   0          2m14s
   pod/webpage-deployment-9bff68bcf-tqcck   1/1     Running   0          2m14s
   
   NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
   service/webpage-service   ClusterIP   10.96.132.13   <none>        8080/TCP   2m14s
   
   NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/webpage-deployment   3/3     3            3           2m14s
   
   NAME                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/webpage-deployment-9bff68bcf   3         3         3       2m14s
   
   NAME                                        CLASS    HOSTS   ADDRESS          PORTS   AGE
   ingress.networking.k8s.io/webpage-ingress   <none>   *       146.56.118.147   80      2m13s
   

10. If you access the webpage through the Ingress address, you can see that the Blue app is currently working well.

    • Address example) http://146.56.118.147/webpage

    

Test - Deploy new version - Green version

1. Go back to the source and change the code (e.g., change the environment variable of Dockerfile) and apply the code by Commit & Push

   ...
   ENV VERSION="2.1"
   ENV MESSAGE="Hello OCI DevOps Canary"
   ENV BACKGROUND="green"
   ...
   

2. Rebuild, wait for the deployment pipeline to run and stop at approval-stage.

3. Check the Kubernetes deployment status.

   You can see that a new version has been deployed to ns-canary-stage.

   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get pod -n ns-canary-stage
   NAME                                  READY   STATUS    RESTARTS   AGE
   webpage-deployment-6d8b64947b-2hqm6   1/1     Running   0          2m33s
   webpage-deployment-6d8b64947b-dtl99   1/1     Running   0          2m19s
   webpage-deployment-6d8b64947b-rqtpg   1/1     Running   0          2m26s
   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get pod -n ns-canary-prod
   NAME                                 READY   STATUS    RESTARTS   AGE
   webpage-deployment-9bff68bcf-4v962   1/1     Running   0          13m
   webpage-deployment-9bff68bcf-9w7d5   1/1     Running   0          13m
   webpage-deployment-9bff68bcf-tqcck   1/1     Running   0          13m
   

4. If you look at the annotations of webpage-ingress in ns-canary-stage, you can see that canary-weight: has been changed to “25”.

   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get ingress webpage-ingress -n ns-canary-stage -o yaml
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       kubernetes.io/ingress.class: nginx
       nginx.ingress.kubernetes.io/canary: "true"
       nginx.ingress.kubernetes.io/canary-by-header: redirect-to-canary
       nginx.ingress.kubernetes.io/canary-weight: "25"
       nginx.ingress.kubernetes.io/rewrite-target: /$2
   ...
   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get ingress webpage-ingress -n ns-canary-prod -o yaml
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       kubernetes.io/ingress.class: nginx
       nginx.ingress.kubernetes.io/rewrite-target: /$2
   ...
   

5. Access the web page. Requested with a new version in ns-canary-stage with a 25% chance.

   

6. Go back to the running deployment pipeline and approve in approval-stage.

7. Check the Kubernetes deployment status again.

   ns-canary-stage remains the same, and you can see that a new version has been deployed with ns-canary-prod.

   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get pod -n ns-canary-stage
   NAME                                  READY   STATUS    RESTARTS   AGE
   webpage-deployment-6d8b64947b-2hqm6   1/1     Running   0          19m
   webpage-deployment-6d8b64947b-dtl99   1/1     Running   0          18m
   webpage-deployment-6d8b64947b-rqtpg   1/1     Running   0          19m
   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get pod -n ns-canary-prod
   NAME                                  READY   STATUS    RESTARTS   AGE
   webpage-deployment-6d8b64947b-fxnf8   1/1     Running   0          117s
   webpage-deployment-6d8b64947b-lm44m   1/1     Running   0          2m4s
   webpage-deployment-6d8b64947b-zv8p8   1/1     Running   0          111
   

8. If you look at the annotations of webpage-ingress in ns-canary-stage, you can see that canary-weight: has been changed to “0”.

   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get ingress webpage-ingress -n ns-canary-stage -o yaml
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       kubernetes.io/ingress.class: nginx
       nginx.ingress.kubernetes.io/canary: "true"
       nginx.ingress.kubernetes.io/canary-by-header: redirect-to-canary
       nginx.ingress.kubernetes.io/canary-weight: "0"
       nginx.ingress.kubernetes.io/rewrite-target: /$2
   ...
   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get ingress webpage-ingress -n ns-canary-prod -o yaml
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       kubernetes.io/ingress.class: nginx
       nginx.ingress.kubernetes.io/rewrite-target: /$2
   ...
   

9. Access the web page. Now the new version is only requested towards the deployed ns-canary-prod.

   On the ns-canary-stage side, there is no request even if you check the access log.

   

   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl logs -lapp=webpage -n ns-canary-stage -f
   
   ...
   

Test - Deploy Additional Changes - Red Deploy

1. Go back to the source and change the code (e.g., change the environment variable of Dockerfile) and apply the code by Commit & Push

   ...
   ENV VERSION="2.2"
   ENV MESSAGE="Hello OCI DevOps Canary"
   ENV BACKGROUND="red"
   ...
   

2. Rebuild, wait for the deployment pipeline to run and stop at approval-stage.

3. Check the Kubernetes deployment status.

   You can see that a new version has been deployed to ns-canary-stage.

   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get pod -n ns-canary-stage
   NAME                                  READY   STATUS    RESTARTS   AGE
   webpage-deployment-79fd4f869b-2kvn4   1/1     Running   0          98s
   webpage-deployment-79fd4f869b-khbvk   1/1     Running   0          110s
   webpage-deployment-79fd4f869b-vwzmn   1/1     Running   0          91s
   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get pod -n ns-canary-prod
   NAME                                  READY   STATUS    RESTARTS   AGE
   webpage-deployment-6d8b64947b-fxnf8   1/1     Running   0          23m
   webpage-deployment-6d8b64947b-lm44m   1/1     Running   0          23m
   webpage-deployment-6d8b64947b-zv8p8   1/1     Running   0          23m
   

4. If you look at the annotations of webpage-ingress in ns-canary-stage, you can see that canary-weight: has been changed to “25”.

   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get ingress webpage-ingress -n ns-canary-stage -o yaml
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       kubernetes.io/ingress.class: nginx
       nginx.ingress.kubernetes.io/canary: "true"
       nginx.ingress.kubernetes.io/canary-by-header: redirect-to-canary
       nginx.ingress.kubernetes.io/canary-weight: "25"
       nginx.ingress.kubernetes.io/rewrite-target: /$2
   ...
   

5. Access the web page. Requested with a new version in ns-canary-stage with a 25% chance.

6. After approval in approval-stage, the new version is distributed to ns-canary-prod in the same way as before, and all requests are forwarded to the new version.

7. If the approval-stage rejects, the weight on the ns-canary-stage side is changed to canary-weight: “0”, and the request for the test version is stopped.

   

   [opc@jumpbox-945115 oci-devops-oke-webpage]$ kubectl get ingress webpage-ingress -n ns-canary-stage -o yaml
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       kubernetes.io/ingress.class: nginx
       nginx.ingress.kubernetes.io/canary: "true"
       nginx.ingress.kubernetes.io/canary-by-header: redirect-to-canary
       nginx.ingress.kubernetes.io/canary-weight: "0"
       nginx.ingress.kubernetes.io/rewrite-target: /$2
   ...
   

test - rollback

If a problem occurs while servicing the new version, rollback to the previous version is available.

1. Navigate to the deployment pipeline that deployed the current version in the deployment history.

2. From the Traffic Shift menu, click Revert traffic shift.

   

3. Choose what to restore during history.

   • Choose from history

     

   • I want to know which one to choose, but let’s choose first.

     

4. Manual rollback operation is complete.

   

5. When you access the webpage, you can see that you have returned to the previous app.