6.3 Installing Multicluster Verrazzano

reference document

• https://verrazzano.io/latest/docs/applications/multicluster/

Install Verrazzano

Install Verrazzano on admin-cluster

Installing Verrazzano in a single cluster environment is the same as installing it. dev or prod Install as profile. However, the environment name is specified in the installation yaml file (eg, install-oci.yaml) to distinguish it from managed-cluster.

• example

  ...
  spec:
    environmentName: admin
    profile: dev
    ...
  

Install Verrazzano on managed-cluster

Installing Verrazzano in a single cluster environment is the same as installing it. Install with the managed-cluster profile. However, the environment name is specified in the installation yaml file (eg, install-oci.yaml) to distinguish it from managed-cluster.

• example

  ...
  spec:
    environmentName: mc-1
    profile: managed-cluster
    ...
  

1. Check the installation result

   kubectl get verrazzano my-verrazzano -o yaml
   

2. Example results

   • With the managed-cluster profile, only the components for Managed Cluster are installed.

     status:
       components:
         cert-manager:
         ...
       instance:
         consoleUrl: https://verrazzano.mc-1.thekoguryo.ml
         prometheusUrl: https://prometheus.vmi.system.mc-1.thekoguryo.ml
       state: Ready
       version: 1.1.0
   

Register Managed Cluster

preparatory steps

Admin Cluster configuration

1. Context change

   $ kubectl config use-context admin-cluster
   Switched to context "admin-cluster".
   $ kubectl config get-contexts
   CURRENT   NAME                CLUSTER               AUTHINFO           NAMESPACE
   *         admin-cluster       cluster-cbrgetu5uwa   user-cbrgetu5uwa
             managed-cluster-1   cluster-cpcnsgcmqsq   user-cpcnsgcmqsq
             managed-cluster-2   cluster-cvoyziqrrzq   user-cvoyziqrrzq
   

2. Check the API Server address

   $ kubectl cluster-info
   Kubernetes control plane is running at https://129.154.60.75:6443
   

3. Create a ConfigMap with the verified address

   kubectl apply -f <<EOF -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: verrazzano-admin-cluster
     namespace: verrazzano-mc
   data:
     server: "https://129.154.60.75:6443"
   EOF
   

Registration Steps - Based on LetsEncrypt staging

Communication between Admin Verrazzano and the Managed Cluster is over https. In addition to known CA certificates, when using Self-Signed certificates and Let’s Encrypt Staging, the CA certificate of the Managed Cluster must be registered in the Admin Cluster for smooth communication between the Admin Cluster and the Managed Cluster. For related procedures, please refer to the document below.

• Refer to related documents if necessary - https://verrazzano.io/latest/docs/setup/install/multicluster/#prerequisites

※ In the case of Let’s Encrypt Staging, additional work is required due to the client environment that is not updated due to the expiration of the DST Root CA.

• Let’s Encrypt Staging
  • https://letsencrypt.org/docs/staging-environment/

Admin Cluster configuration

1. Context change

   kubectl config use-context admin-cluster
   

2. Register LetsEncrypt staging CA

   # (STAGING) Artificial Apricot R3
   curl -O https://letsencrypt.org/certs/staging/letsencrypt-stg-int-r3.pem
   # (STAGING) Pretend Pear X1
   curl -O https://letsencrypt.org/certs/staging/letsencrypt-stg-root-x1.pem
   MGD_CA_CERT=$(cat *.pem)
   echo $MGD_CA_CERT
   
   kubectl create secret generic "ca-secret-managed-cluster-1" \
     -n verrazzano-mc \
     --from-literal=cacrt="$MGD_CA_CERT" \
     --dry-run=client \
     -o yaml > ca-secret-managed-cluster-1.yaml
   
   kubectl apply -f ca-secret-managed-cluster-1.yaml
   

3. Register as Managed Cluster

   • If the registered cluster name is managed-cluster-1

   kubectl apply -f <<EOF -
   apiVersion: clusters.verrazzano.io/v1alpha1
   kind: VerrazzanoManagedCluster
   metadata:
     name: managed-cluster-1
     namespace: verrazzano-mc
   spec:
     description: "VerrazzanoManagedCluster object"
     caSecret: ca-secret-managed-cluster-1
   EOF
   

4. Waiting for registration

   kubectl wait --for=condition=Ready \
       vmc managed-cluster-1 -n verrazzano-mc
   

5. Create registration file for managed cluster

   kubectl get secret verrazzano-cluster-managed-cluster-1-manifest \
       -n verrazzano-mc \
       -o jsonpath={.data.yaml} | base64 --decode > managed-cluster-1-register.yaml
   

Configure Managed Cluster

1. Context change

   kubectl config use-context managed-cluster-1
   

2. Registration

   kubectl apply -f managed-cluster-1-register.yaml
   

3. Registration result

   When the Managed Cluster is registered to the Admin Cluster, the Managed Cluster periodically updates the status of the resource (managed-cluster-1) registered as a VerrazzanoManagedCluster type prior to the Admin Cluster.

4. Check for ranch agent errors

   POD does not start due to certificate error x509: certificate signed by unknown authority as shown below.

   [opc@bastion-host ~ (⎈ |managed-cluster-1:default)]$ kubectl get pod -n cattle-system
   NAME                                    READY   STATUS             RESTARTS   AGE
   cattle-cluster-agent-5dd4dd9594-v9tf8   0/1     CrashLoopBackOff   3          2m8s
   [opc@bastion-host ~ (⎈ |managed-cluster-1:default)]$ kubectl logs -n cattle-system cattle-cluster-agent-5dd4dd9594-v9tf8
   ...
   INFO: https://rancher.admin.thekoguryo.ml/ping is accessible
   ...
   time="2022-01-04T06:03:57Z" level=info msg="Certificate details from https://rancher.admin.thekoguryo.ml"
   time="2022-01-04T06:03:57Z" level=info msg="Certificate #0 (https://rancher.admin.thekoguryo.ml)"
   time="2022-01-04T06:03:57Z" level=info msg="Subject: CN=*.admin.thekoguryo.ml"
   time="2022-01-04T06:03:57Z" level=info msg="Issuer: CN=(STAGING) Artificial Apricot R3,O=(STAGING) Let's Encrypt,C=US"
   time="2022-01-04T06:03:57Z" level=info msg="IsCA: false"
   time="2022-01-04T06:03:57Z" level=info msg="DNS Names: [*.admin.thekoguryo.ml]"
   time="2022-01-04T06:03:57Z" level=info msg="IPAddresses: <none>"
   time="2022-01-04T06:03:57Z" level=info msg="NotBefore: 2022-01-04 04:52:39 +0000 UTC"
   time="2022-01-04T06:03:57Z" level=info msg="NotAfter: 2022-04-04 04:52:38 +0000 UTC"
   ..
   time="2022-01-04T06:03:57Z" level=info msg="Certificate #1 (https://rancher.admin.thekoguryo.ml)"
   time="2022-01-04T06:03:57Z" level=info msg="Subject: CN=(STAGING) Artificial Apricot R3,O=(STAGING) Let's Encrypt,C=US"
   time="2022-01-04T06:03:57Z" level=info msg="Issuer: CN=(STAGING) Pretend Pear X1,O=(STAGING) Internet Security Research Group,C=US"
   ...
   time="2022-01-04T06:03:57Z" level=info msg="Certificate #2 (https://rancher.admin.thekoguryo.ml)"
   time="2022-01-04T06:03:57Z" level=info msg="Subject: CN=(STAGING) Pretend Pear X1,O=(STAGING) Internet Security Research Group,C=US"
   time="2022-01-04T06:03:57Z" level=info msg="Issuer: CN=(STAGING) Doctored Durian Root CA X3,O=(STAGING) Internet Security Research Group,C=US"
   ...
   time="2022-01-04T06:03:57Z" level=fatal msg="Certificate chain is not complete, please check if all needed intermediate certificates are included in the server certificate (in the correct order) and if the cacerts setting in Rancher either contains the correct CA certificate (in the case of using self signed certificates) or is empty (in the case of using a certificate signed by a recognized CA). Certificate information is displayed above. error: Get \"https://rancher.admin.thekoguryo.ml\": x509: certificate signed by unknown authority"
   

5. Modify and reflect cattle-cluster-agent.

   • correction

     kubectl edit deploy cattle-cluster-agent -n cattle-system
     

   • More details

     Download the new certificate separately via initContainer and pass it to cluster-register container via shared volume.

   apiVersion: apps/v1
   kind: Deployment
   ...
         containers:
         - env:
           ...
           name: cluster-register
           ...
           volumeMounts:
           - mountPath: /cattle-credentials
             name: cattle-credentials
             readOnly: true
           - mountPath: /etc/pki/tls/certs/
             name: certs
         initContainers:
         - name: prepare-to-fix-ca-expire
           image: ghcr.io/verrazzano/rancher-agent:v2.5.9-20211209021347-2e57ce2a4
           command:
           - /bin/sh
           - -c
           - |
             echo start
             curl -k https://letsencrypt.org/certs/staging/letsencrypt-stg-root-x1.pem -o /etc/pki/ca-trust/source/anchors/letsencrypt-stg-root-x1.pem
             update-ca-trust
             cp /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /certs/ca-bundle.crt
             cp /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt /certs/ca-bundle.trust.crt
             echo end          
           volumeMounts:
           - mountPath: /certs/
             name: certs   
   ...
         volumes:
         - name: cattle-credentials
           secret:
             defaultMode: 320
             secretName: cattle-credentials-2b150a3
         - name: certs
           emptyDir: {}
   status
   ...
   

   • Recheck the result

   [opc@bastion-host ~ (⎈ |managed-cluster-1:default)]$ kubectl get pod -n cattle-system
   NAME                                    READY   STATUS    RESTARTS   AGE
   cattle-cluster-agent-6b785ffd86-vshsw   1/1     Running   0          42s
   

   • When the above POD starts normally, you can check that the Managed Cluster is registered as Active in Rancher of Admin Cluster.

Check registration result

1. Change context to admin cluster

   kubectl config use-context admin-cluster
   

2. Check the current status of the previously registered Managed Cluster resources

   • The value of status.lastAgentConnectTime confirms that the connection was made using the last connected time. You can see that managed-cluster-1’s apiUrl and prometheusHost address are also registered.

   $ kubectl get vmc managed-cluster-1 -n verrazzano-mc -o yaml
   apiVersion: clusters.verrazzano.io/v1alpha1
   kind: VerrazzanoManagedCluster
   metadata:
     ...
   spec:
     ...
   status:
     apiUrl: https://verrazzano.mc-1.thekoguryo.ml
     conditions:
     - lastTransitionTime: "2021-12-31T06:48:59Z"
       message: Ready
       status: "True"
       type: Ready
     lastAgentConnectTime: "2021-12-31T06:57:06Z"
     prometheusHost: prometheus.vmi.system.mc-1.thekoguryo.ml
     rancherRegistration:
       message: Registration of managed cluster completed successfully
       status: Completed
     state: Active
   

final check

Promethus

1. Connect to Prometheus of Verrazzano Admin Cluster.

   • Ex) https://prometheus.vmi.system.admin.thekoguryo.ml/

2. If you use the sample query node_disk_io_time_seconds_total, you can see that the metric information of the Verrazzano Managed Cluster is also queried as shown in the figure.

   

Kibana

1. Connect to Kibana of Verrazzano Admin Cluster.

   • Example) https://kibana.vmi.system.admin.thekoguryo.ml/

2. If there is no app deployed yet, create an index pattern based on the namespace (eg, cert-manager) in all Verrazzano Clusters.

3. Click cluster_name in the selectable fields. As shown below, you can see that logs are also being pulled from the Admin Cluster (local) and the registered Managed Cluster.

   

Rancher

1. Connect to Prometheus of Verrazzano Admin Cluster.

   • Example) https://rancher.admin.thekoguryo.ml/

2. You can see that Managed Cluster is normally registered as shown below.

   

3. You can see that it looks good on the screen as a cluster by clicking Explorer.