Metricbeat is a lightweight shipper that collects and forwards metrics from your systems and services to Elasticsearch or Logstash. It provides valuable insights into the health and performance of your infrastructure, making it a key tool for monitoring and observability. By the end of this tutorial, you will have Metricbeat up and running, allowing you to monitor your clusters effectively.
Creation of a Storage Account and Container to export files with the collection of K8s metrics. #
Create a Storage Account in Azure: #
Example using Azure CLI:
az storage account create -n <NEW_STORAGE_ACCOUNT_NAME> -g <RESOURCE_GROUP_NAME> -l <REGION> --sku Standard_LRS --kind StorageV2 
NOTE: The “Allow storage account key access” setting must be enabled for the integration to work.
Create a container in the Storage Account for file integration. #
Example using Azure CLI:
az storage container create --name ${STORAGE_ACCOUNT_CONTAINER_NAME} --account-name <NEW_STORAGE_ACCOUNT_NAME> 
Obtain the Storage Account credentials #
Obtain the Storage Account Access Key to configure the integration with metricbeat
Example using Azure CLI:
az storage account keys list -g <RESOURCE_GROUP_NAME> -n <NEW_STORAGE_ACCOUNT_NAME> --query [1].value In this example, we managed to obtain the first active access key for the integration
Create the Secret in the AKS cluster with the Storage Account integration settings #
Example using kubectl:
kubectl create secret generic <SECRET_NAME> --from-literal
azurestorageaccountname=<NEW_STORAGE_ACCOUNT_NAME> --from-literal azurestorageaccountkey=<ACCESS_KEY> --type=Opaque - <SECRET_NAME>: Name of the new secret that will be created in the AKS cluster
- <NEW_STORAGE_ACCOUNT_NAME>: Configured in step 1.1
- <ACCESS_KEY>: Obtained in step 1.3
Enable integration add-on in the AKS cluster. #
Add-on Azure Blob storage Container Storage Interface (CSI) driver no cluster #
Example using Azure CLI:
az aks update --enable-blob-driver -n <CLUSTER_NAME> -g <RESOURCE_GROUP_NAME> -y Configure the metricbeat deployment with the export of files to the Storage Acount Container. #
Deployment of kube-state-metrics #
Get the kube-state-metrics template and deploy it:
https://kube-state-metrics-template.s3.amazonaws.com/kube-state-metrics-template.yml
Deployment of metricbeat #
Get the metricbeat template:
Manually adjust the following parameters in the template. Adjust the parameter:
--subdirectory=aks/<RESOURCE_GROUP_NAME>/<YOUR_REGION>/<CLUSTER_NAME>/ - <RESOURCE_GROUP_NAME>: name of the resource group that the AKS cluster is in <YOUR_REGIAO>: region that the AKS cluster is in
- <CLUSTER_NAME>: name of the cluster that metricbeat will collect and send the metrics in the integration.
- <STORAGE_ACCOUNT_NAME>: Name of the Storage Account configured in the step 1.1.
- <CONTAINER_NAME_IN_STORAGE_ACCOUNT>: Container name configured in step 1.2.
This template is already prepared for creating objects in the cluster for metricbeat to work:
- ServiceAccount – will be used when running the metricbeat service;
- ClusterRole – k8s API settings and objects – read only;
- Roles and ClusterRoleBinding – complementary configurations for reading the k8s APIs in metricbeat;
- ConfigMaps – parameters and configurations for metricbeat integration with kubernetes;
- DaemonSet – metricbeat service that collects metrics and exports files to the Storage Account Container.
Deploy metricbeat and check export. #
Continue with the deployment of metricbeat in the cluster after applying the configurations. After deployment, it is important to check whether the component is collecting metrics and exporting them to the integration Storage Account container.
Check if the metricbeat pods are running. #
Example:
kubectl get pods -n kube-system -o wide 
NOTE: metricbeat will upload one pod per node to collect metrics
Check the pod logs to see if metrics collection events are being generated. #
Example:
Check that after a few minutes of the pod running, if the files are being exported to the integration Storage Account Container: #
Example:
- <NEW_CONTAINER_NAME> : name of the container inserted in the subdirectory as shown in 3.2
- <NOME_DO_RESOURCE_GROUP>: region inserted in the subdirectory as shown in 3.2
- <YOUR_REGION>: region inserted in the subdirectory as shown in 3.2
- <CLUSTER_NAME>: name of the cluster configured in the step 3.2
File export #
Due to metricbeat limitations, only 1024 log files are preserved. For the system to function correctly, at least files from the last 7 days must be preserved – we recommend, however, that they be kept for at least 35 days.
As the available configuration is by size and not by time, we recommend the following:
- Leave the default setting (which is 10mb per file) for 1 day;
- After exactly 24 hours, check the number of files generated:
- If more than 145 files were generated, please inform us as the Container will not retain files for a week;
- If 29 or more were generated, your configuration is adequate;
- If it is less than 29, apply the following formula:
FILE_SIZE = 10240 / 29 * QUANTITY For example, if 5 files were generated:
FILE_SIZE = 10240 / 29 * 5 = 1765 Therefore, within the metricbeat-deployment-template-aks-blob-csi.yml file, configure data -> metricbeat.yml: -> output.file -> rotate_every_kb value of 1765 instead of 10240.
