5.2.1 Log monitoring using OCI Logging

Application log monitoring deployed on OKE can be monitored through OCI Logging service.

Set OCI Logging service permission

Create a Dynamic Group for Worker Nodes

1. Log in to the OCI console.

2. Go to Identity & Security > Identity > Compartments from the top left hamburger menu.

3. Check the OCID of the Compartment with the OKE cluster.

   

4. Go to the left Dynamic Group menu and create a Dynamic Group with the following rules.

   instance.compartment.id = '<compartment-ocid>'
   

   

Granting OCI Logging service permission for Dynamic Group

1. Go to the Policy menu on the left and create a policy with the following rules. Create a Policy for the Dynamic Group you just created.

   allow dynamic-group <dynamic-group-name> to use log-content in compartment <compartment-name>
   

   

Set up custom log for container

Create Log Group

Log Group is literally a unit of log group that manages logs. To create a custom log, we first create it.

1. Go to Observability & Management > Logging > Log Groups from the top left hamburger menu.

2. Click Create Log Group to create the log group.

   

Create Custom Log

Custom Logs are mappings to logs collected by custom applications. Define a Custom Log, and define a log collector for it.

1. Go to the Resources > Logs menu and click Create custom log.

2. Create custom log by setting log name and retention period.

   

3. Agent is set separately for better understanding. For now, select Add configuration later

   

Agent Configuration Settings

Agent Configuration is the part that sets the agent that collects logs.

1. Go to the Logging > Agent Configurations menu and click Create agent log.

2. Specify the Agent Name and Target Host Group as the Dynamic Group created earlier.

   

3. In the Agent setting section, specify the path where the log is located and the delivery location of the collected logs.

   • log input: /var/log/containers/*.log

     VM in the Dynamic Group specified earlier, here, specify the location of the log to collect on the OKE cluster Worker Node VM.

   • log destination: Specify the name of the custom log created earlier to forward the collected logs.

   

Note: Container log location on Worker Node VM

1. Environment where SSH access to Worker Node VM is possible, e.g.) When connecting to Worker Node from bastion host, the container log location is as follows.

   [opc@bastion-host ~]$ ssh opc@10.0.10.175
   Last login: Tue Nov 16 06:43:43 2021 from bastion-host.suba22926d1b.okecluster1.oraclevcn.com
   [opc@oke-cjy33uzxhyq-nkglpmyewja-siouvw2ymqq-2 ~]$ sudo su
   [root@oke-cjy33uzxhyq-nkglpmyewja-siouvw2ymqq-2 opc]# cd /var/log/containers/
   [root@oke-cjy33uzxhyq-nkglpmyewja-siouvw2ymqq-2 containers]# ls -la
   total 16
   drwxr-xr-x.  2 root root 4096 Nov 15 14:05 .
   drwxr-xr-x. 13 root root 4096 Nov 15 13:55 ..
   lrwxrwxrwx.  1 root root  100 Nov 15 13:52 coredns-c5d4bf466-qdgrh_kube-system_coredns-a533d7375a4bd90b894d533e637bae6ce38e2e7d89cd0ff16e34bd120111c7e4.log -> /var/log/pods/kube-system_coredns-c5d4bf466-qdgrh_24b226a0-0fb3-4ede-b02f-17b177e6c248/coredns/0.log
   ...
   lrwxrwxrwx.  1 root root  101 Nov 15 14:05 nginx-bvs-pvc-7b574c9c5c-vdpdh_default_nginx-cc996c8fd2281d1bec7fdad75518c66a2ec6f03adc214f8bbd97c26967e8d0e7.log -> /var/log/pods/default_nginx-bvs-pvc-7b574c9c5c-vdpdh_aab712c6-81e1-434c-b051-f3e8fe50fa24/nginx/0.log
   lrwxrwxrwx.  1 root root  106 Nov 15 13:52 proxymux-client-cb86k_kube-system_proxymux-client-471c1c80fe2dc01c5e1413819af58d968796541cc744384f1b2daa566873d8ba.log -> /var/log/pods/kube-system_proxymux-client-cb86k_153b7b7a-a777-4be9-b971-657eb8ecdddd/proxymux-client/0.log
   

2. Since /var/log/containers/*.log is a link as shown above, if you want to use the kubernetes namespace as a standard, set the log path to the default namespace /var/log/pods/default_*/*/*.log You can do this.

Logging test

1. To check the application log, try accessing the nginx app distributed as a sample in the previous guide.

   

2. The generated POD log is as follows.

   oke_admin@cloudshell:~ (ap-seoul-1)$ kubectl logs nginx-bvs-pvc-7b574c9c5c-vdpdh -f
   ...
   10.244.0.128 - - [16/Nov/2021:08:10:46 +0000] "GET /?customlogtest HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36" "10.179.87.76"
   

3. If you look up the log even on the Worker Node VM, you can see that the same log occurs in the /var/log/containers/*.log location.

   [root@oke-cjy33uzxhyq-nkglpmyewja-siouvw2ymqq-2 containers]# pwd
   /var/log/containers
   [root@oke-cjy33uzxhyq-nkglpmyewja-siouvw2ymqq-2 containers]# tail -f nginx-bvs-pvc-7b574c9c5c-vdpdh_default_nginx-cc996c8fd2281d1bec7fdad75518c66a2ec6f03adc214f8bbd97c26967e8d0e7.log
   ...
   2021-11-16T08:10:46.961696444+00:00 stdout F 10.244.0.128 - - [16/Nov/2021:08:10:46 +0000] "GET /?customlogtest HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36" "10.179.87.76"
   

4. In the OCI service console, go back to the Logging screen.

5. Agent, Logs, Log Groups In each screen, you can search the log in Resources > Explore Log. To search, click Explore with Log Search at the top right of the log list.

   

6. In the Custom filters item, you can search by search value such as POD name or customlogtest in the test URL above. Enter a value in Custom filters and hit enter.

   • It may take 5 minutes or so as there is a collection cycle through the Log Agent.
   • If you have accessed the MuShop app test URL only once, you may not be able to view the log even if you keep waiting because the log flush does not occur within the section.

   

7. You can check the searched log data.