How to install the Metricbeat component in OKE

Metricbeat is a lightweight agent that collects and sends metrics from your systems and services to Elasticsearch or Logstash. It provides valuable insights into the health and performance of your infrastructure , making it an essential tool for monitoring and observability. By the end of this tutorial, you will have Metricbeat set up and running, allowing you to effectively monitor your OKE clusters.

Policies  #

Verify that the following permissions are properly configured at the tenancy level:

ALLOW any-user to manage file-family in tenancy where request.principal.type = 'cluster'
ALLOW any-user to use virtual-network-family in tenancy where request.principal.type = 'cluster'
ALLOW any-user to manage file-family in tenancy where request.principal.type = 'cluster'
ALLOW any-user to use virtual-network-family in tenancy where request.principal.type = 'cluster'

File System Integration #

New or existing File System #

Reference : https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingpersistentvolumeclaim_Provisioning_PVCs_on_FSS.htm

Get the subnet of the K8s API endpoint configured on the cluster.

Example:

Create a new File System with a Mount Target or create a new Mount Target in the same subnet as the K8s API endpoint configured in the cluster .

Example:

File System -> Mount Target: mount-target-metricbeat-doc-tests

Create an Export Path on the Mount target in the following format: #

/oke/<SUA_REGIAO>/<NOME_DO_COMPARTMENT>/<NOME_DO_CLUSTER>

• <YOUR_REGION> : cluster region
• <COMPARTMENT_NAME> : name of the compartment where the cluster is located
• <CLUSTER_NAME> : cluster name

Example:

Configure security list on subnet to allow File System #

Open TCP and UDP ports for 111 and 2048-2050 on Ingress and Egress in the same subnet as the K8s API endpoint configured in the cluster.

Example:

Configure metricbeat deployment with file export to File System #

kube-state-metrics deployment #

Get the kube-state-metrics template and deploy it:

https://kube-state-metrics-template.s3.amazonaws.com/kube-state-metrics-template.yml

Metricbeat Deployment #

Get the Metricbeat template:

https://metricbeat-deployment-template-oke-fss.s3.amazonaws.com/metricbeat-deployment-fss-csi-driver.yml

Manually adjust the following parameters in the template:

<ocid1.filesystem.oc1.[region].[id]>:<mountTarget IP>:/oke/<SUA_REGIAO>/<NOME_DO_COMPARTMENT>/<NOME_DO_CLUSTER>

• <ocid1.filesystem.oc1.[region].[id]> -> ID of the created or existing File System, as per step 2.1
• <mountTarget IP> -> Internal IP generated by creating the Mount Target as per step 2.1
• :/oke/<YOUR_REGION>/<COMPARTMENT_NAME>/<CLUSTER_NAME> -> folder for export configured in Mount Target as per step 2.2 .

This template is already prepared for creating objects in the cluster for Metricbeat to work:

• Persistent Volume and Persistent Volume Claim – previously parameterized with the File System for integration
• ServiceAccount – will be used when executing the metricbeat service;
• ClusterRole – k8s API and object configurations – read-only;
• Roles and ClusterRoleBinding – additional configurations for reading k8s APIs in metricbeat;
• ConfigMaps – parameters and configurations for integrating metricbeat with kubernetes;
• DaemonSet – metricbeat service that collects metrics and exports files to the File System

NOTE : OKE clusters from versions 1.28 onwards have the fss.csi.oraclecloud.com driver already available to be used in metricbeat deployment.

Deploy Metricbeat and verify the export.  #

Continue with the deployment of Metricbeat in the cluster after applying the configurations. After
deployment, it is important to check if the component is collecting the metrics and exporting them to the File System export path:

Verify that Metricbeat pods are running.  #

Example: 

kubectl get pods -n kube-system -o wide 

NOTE  : Metricbeat will upload one pod per node to collect metrics

Check the pod logs to see if metrics collection events are being generated.  #

Example: 

Verify that after a few minutes of pod running, files are being exported to the File System export path:  #

Below is an example of checking the files exported from within a Compute instance with mapping to the folder with the File System export path configured in step 2.1 .

We can see these mapping instructions in “Mount commands”:

Accessing and listing the exported files from an instance as an example:

File export  #

Due to Metricbeat limitations, only 1024 log files are preserved. For the system to function properly, at least the last 7 days of log files must be preserved – we recommend, however, that they be kept for at least 35 days. 

Since the available configuration is by size and not by time, we recommend the following: 

• Leave the default setting (which is 10 MB per file) for 1 day;
• After exactly 24 hours, check the number of files generated: 
• If more than 145 files were generated, please let us know as the bucket will not retain files for a week; 
• If 29 or more files were generated, your configuration is fine;
• If the number of files is less than 29, apply the following formula: 

FILESIZE = 10240 / 29 * QUANTITY 

For example, if 5 files were generated: 

FILESIZE = 10240 / 29 * 5 = 1765 

So inside metricbeat-deployment-template-eks-s3.yml file, set data -> metricbeat.yml: -> output.file ->  rotate_every_kb  with value of 1765 instead of 10240.