5.2.2 Log monitoring using OCI Search service

OpenSearch and OpenDashboards will be forked from ElasticSearch and Cabana in 2021 and are operating as separate open source projects. OCI OpenSearch is a managed service provided by OCI.

Setting policy to use OCI Search service

Granting OCI Search service permission

Allow service opensearch to manage vnics in compartment <NETWORK_RESOURCES_COMPARTMENT>
Allow service opensearch to manage vcns in compartment <NETWORK_RESOURCES_COMPARTMENT>
Allow service opensearch to use subnets in compartment <NETWORK_RESOURCES_COMPARTMENT>
Allow service opensearch to use network-security-groups in compartment <NETWORK_RESOURCES_COMPARTMENT>

Granting OCI Search Service Management Permissions in Oracle Cloud Console

• Example when the group the user belongs to is SearchOpenSearchAdmins

Allow group SearchOpenSearchAdmins to manage opensearch-family in compartment <CLUSTER_RESOURCES_COMPARTMENT>

Create an OCI OpenSearch cluster

1. Log in to the Oracle Cloud Console.

2. Go to Databases > OpenSearch > Clusters from the top left hamburger menu.

3. Click the Create Cluster button to create a cluster.

4. Select the compartment location, name, and version to create.

   

5. Select the node configuration to create. There are three possible configurations, here we choose the Development configuration.

   • Development: 1 Master Node, 1 Data Node, 1 OpenSearch Dashboard Node
   • Application Search, Log Analytics: 3 Master Nodes, 3 Data Nodes, 3 OpenSearch Dashboard Nodes
   • After selecting the configuration, all node sizes can be modified. You can also change the size after cluster creation.

   

6. Set the subnets the cluster will use. Here, we have chosen the VCN and subnet already created when creating the OKE cluster for convenience.

   

7. Confirm the configuration information to select and create the cluster.

8. Based on the test date based on the development configuration, the creation time took about 20 minutes.

9. Created Cluster Information

   • As you can see from the cluster information, both OpenSearch and OpenSearch Dashboard have only Private IP and Private FQDN. As of mid-October 2022, public IP is not yet supported.

   

10. Check the OpenSearch API Endpoint, Private IP of the cluster and the API Endpoint and Private IP of the OpenSearch Dashboard.

    • As of September 15, 2022, the endpoint domain has been changed from oci.oracleiaas.com to oci.oraclecloud.com. And the certificate was also changed from Self-Signed certificate to DigiCert certificate. So, you no longer need to add a self-signed certificate to the client when calling the API.

      • See Note in [OCI Documentation > Search Service with OpenSearch > Connection to a Cluster] (https://docs.oracle.com/en-us/iaas/Content/search-opensearch/Tasks/ingestingociopensearchdata.htm#connecttocluster)

    • If the cluster or endpoint domain created before September 15, 2022 ends with oci.oracleiaas.com, refer to the previous document for the integration procedure.

      • [5.2.2 Log monitoring using OpenSearch based OCI Search service - Self-Signed](../../../../archives/oracle-cloudnative/oci-services/logging/2.oci-opensearch- self-signed/)

Setting up to connect to the cluster

Create a JumpBox VM

OpenSearch Dashboard does not currently support Public IP. Create additional JumpBox VM with public IP to access OpenSearch Dashboard.

1. In Oracle Cloud Console, go to Compute > Instances from the top left hamburger menu.

2. Create a new Compute VM.

   • Name: ex) jumpbox

   • Networking: Select a VCN or Public Subnet that can access the created OpenSearch cluster and assign a public IP.

     Ex) In this case, we used the same network as the OpenSearch cluster we created.

3. Add a security rule so that the jumpbox VM can access OpenSearch from the jumpbox VM.

   • Ingress Rules
     • In the example below, if the jumpbox is located in the subnet 10.0.20.0/24

   Stateless	Source	IP Protocol	Source Port Range	Destination Port Range	Description
   No	0.0.0.0/0	TCP	All	22	SSH
   No	10.0.20.0/24	TCP	All	5601	OpenSearch Dashboard
   No	10.0.20.0/24	TCP	All	9200	OpenSearch API

OpenSearch Dashboard Connection Check

1. Access the OpenSearch Dashboard through the jumpbox VM through SSH tunneling.

   • Based on OpenSearch Cluster information, if OpenSearch Dashboard private ip is 10.0.20.218

   ssh -L 5601:10.0.20.218:5601 opc@jumpbox
   

2. Since SSH tunneling is done, check the connection to the OpenSearch Dashboard with a browser locally.

   • Access address: https://localhost:5601
   • Choose Explore on my own

   

Send log from OKE to OpenSearch

Use OpenSearch and OpenSearch Dashboard instead of ElasticSearch and Kibana in your EFK configuration.

Fluentd configuration

1. Create a Service Account for Fluentd installation and define the relevant permissions.

2. configmap to define additional settings

   • Fluentd-related settings are all defined in a .conf file under /fluentd/etc/ in the container. You can override these files.
   • OCI OpenSearch clusters now use Self Signed certificates by default. Leave only the basics that are absolutely necessary for the connection. If you leave the commented out part and don’t set a value, you will get an error at run time.

3. Define a fluentd damonset

   • Some changes have been made to the YAML in the Fluentd documentation to use the configured configmap.
   • We are changing to use fluent.conf overridden by configmap.
   • You are setting up the Self Signed CA used by the OpenSearch Dashboard.

Install FluentD

1. In fluentd-daemonset-opensearch.yaml, change FLUENT_OPENSEARCH_HOST to the target OpenSearch host.

   ...
           env:
             - name:  FLUENT_OPENSEARCH_HOST
               value: "amaaaaaavsea7yia3enl6q6yrwekrd7qpi3yqaphhcvwowtmzftkt45jel7a.opendashboard.ap-chuncheon-1.oci.oraclecloud.com"
             - name:  FLUENT_OPENSEARCH_PORT
               value: "9200"
             ...
   

2. Install FluentD

   kubectl apply -f fluentd-rbac.yaml
   kubectl apply -f fluentd-configmap-opensearch.yaml
   kubectl apply -f fluentd-daemonset-opensearch.yaml
   

3. Add a security rule to send logs from OKE cluster Worker Node to OpenSearch.

   • Worker Node security rules

     • Target: ex) oke-nodeseclist-quick-oke-cluster-1-04cdcc334

     • Egress Rules:

       The example below is when the OpenSearch location is subnet 10.0.20.0/24.

       Stateless	Destination	IP Protocol	Source Port Range	Destination Port Range	Description
       No	10.0.20.0/24	TCP	All	9200	OpenSearch API

   • Security rules of OpenSearch

     • Target: ex) oke-svclbseclist-quick-oke-cluster-1-04cdcc334

     • Ingress Rules:

       In the example below, if the Worker Node is in the subnet 10.0.10.0/24,

     Stateless	Source	IP Protocol	Source Port Range	Destination Port Range	Description
     No	10.0.10.0/24	TCP	All	9200	OpenSearch API

4. Check the log

   When the fluentd Pod starts up, check the logs to see if it operates normally without any connection errors with OpenSearch.

   $ kubectl get pod -n kube-system
   NAME                                   READY   STATUS    RESTARTS      AGE
   ...
   fluentd-7hbht                          1/1     Running   0             99s
   fluentd-qwrg4                          1/1     Running   0             99s
   fluentd-zfgfz                          1/1     Running   0             99s
   ...
   $ kubectl logs -f fluentd-7hbht -n kube-system -f
   2022-10-13 15:31:30 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
   ...
   2022-10-13 15:31:32 +0000 [info]: adding match pattern="**" type="opensearch"
   ...
   2022-10-13 15:31:32 +0000 [info]: #0 starting fluentd worker pid=14 ppid=6 worker=0
   2022-10-13 15:31:32 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/proxymux-client-s75ds_kube-system_proxymux-client-eca8e5e37a2dbf317be4f25cddcce1f4659c5bf9adecccdf522f1f2317b17dc3.log
   2022-10-13 15:31:32 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/csi-oci-node-vlgnm_kube-system_csi-node-driver-d566fd830f5a575b4d3724946f4a09a23bdf7615527b511a42cbae57c27bab0d.log
   

OpenSearch Dashboard Settings

1. If you have not connected to the jumpbox VM through SSH tunneling, reconnect.

2. Since SSH tunneling is done, check the connection to the OpenSearch Dashboard with a browser locally.

   • Access address: https://localhost:5601

3. In the upper left navigation menu, click OpenSearch Dashboards > Discover.

   

4. Click Create index pattern.

5. Create an index pattern.

   Among the three sources shown below, we use the fluentd source.

   • Index pattern name: fluentd
   • Time field: time

   

   

6. You can see the result of adding the index pattern.

   

7. Click OpenSearch Dashboards > Discover in the upper left navigation menu.

8. You can check the collected logs through the created index pattern.

   • To check the log of the test app, click Add filter to specify namespace_name=default .

   

9. Access the test app.

   • Ex) http://129.154.xxx.xxx/?opensearch-test

10. Check the log

    You can check the log of the test app in the OpenSearch Dashboard as shown below.