Search
Close this search box.
View Categories

How to install Metricbeat component on EKS

5 min read

Metricbeat is a lightweight agent that collects and sends metrics from your systems and services to Elasticsearch or Logstash. It provides valuable insights into the health and performance of your infrastructure , making it an essential tool for monitoring and observability. By the end of this tutorial, you will have Metricbeat set up and running, allowing you to effectively monitor your EKS clusters.

Creating an S3 bucket to export files with K8s metrics collection  #

Create a bucket with default S3 settings:  #

Example: 

AWS new S3 bucket

NOTE: This bucket will store the exported files and EKS cluster metrics collected by the metricbeat component (configurations from steps 3 and 4).

Configure S3 bucket permissions #

Reference : https://docs.aws.amazon.com/eks/latest/userguide/s3-csi.html#s3-create-iam-policy 

Create an IAM policy to gain access to the bucket you created earlier and perform filesystem operations on the bucket.

Policy example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "MountpointBucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::BUCKET_NAME"
      ]
    },
    {
      "Sid": "MountpointFullObjectAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:AbortMultipartUpload",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::BUCKET_NAME/*"
      ]
    }
  ]
}

Enable Integration Add-ons on EKS cluster  #

Add-on Amazon EBS CSI Driver no cluster #

Example using eksctl: 

eksctl create addon \
  --name aws-ebs-csi-driver \
  --cluster <CLUSTER_NAME> \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --version 1.30.0 \
  --force \
  --profile=<YOUR_PROFILE> 


Create the service account and link it with the IAM role for the S3 CSI Driver to work on the cluster #

Reference:  https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html

Example using eksctl:

eksctl create iamserviceaccount \
  --name s3-csi-driver-sa \
  --namespace kube-system \
  --cluster <CLUSTER_NAME> \
  --attach-policy-arn arn:aws:iam::<ACCOUNT_ID>:policy/my-s3-csi-driver-policy \
  --approve \
  --role-name my-s3-csi-driver-role \
  --region <YOUR_REGION> \
  --role-only \
  --profile=<YOUR_PROFILE>

OBS:

  • my-s3-csi-driver-policy : name given to the policy previously created in step 1.2
  • my-s3-csi-driver-role : a name entered in the –role-name parameter , mandatory for the service account to be created with the linked IAM role and with the policy created in step 1.2 attached.

Add-on Mountpoint for Amazon S3 CSI Driver no cluster #

Install version v1.5.1 or higher. Select the IAM role created previously in step 2.2 during installation .

Example using eksctl:

eksctl create addon \
  --name aws-mountpoint-s3-csi-driver \
  --cluster <CLUSTER_NAME> \
  --service-account-role-arn arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME> \
  --version 1.5.1 \
  --force \
  --profile=<YOUR_PROFILE> 

NOTE: <ROLE_NAME> created in step 2.2

Configure Metricbeat deployment with file export to S3 bucket #

Deployment do kube-state-metrics #

Get the kube-state-metrics template and deploy it:

https://kube-state-metrics-template.s3.amazonaws.com/kube-state-metrics-template.yml

Deployment do Metricbeat #

Get the metricbeat template:

https://metricbeat-deployment-template-eks-s3.s3.amazonaws.com/metricbeat-deployment-template-eks-s3.yml

Manually adjust the following parameters in the template:

  • <BUCKET_REGION> : region where the bucket is located
  • <BUCKET_NAME> : name of the bucket for integration and export of files

Adjust the – prefix parameter to: eks/<YOUR_REGION>/<CLUSTER_NAME>/

This template is already prepared for creating objects in the cluster for Metricbeat to work:

  • Persistent Volume and Persistent Volume Claim – previously parameterized with the S3 bucket for integration;
  • ServiceAccount – will be used when executing the metricbeat service;
  • ClusterRole – k8s API and object configurations – read-only;
  • Roles and ClusterRoleBinding – additional configurations for reading k8s APIs in metricbeat;
  • ConfigMaps – parameters and configurations for integrating metricbeat with kubernetes;
  • DaemonSet – metricbeat service that collects metrics and exports files to the S3 bucket.

Deploy Metricbeat and verify the export  #

Proceed with deploying Metricbeat to the cluster after applying the configurations. After deployment, it is important to check if the component is collecting metrics and exporting them to the S3 integration bucket:

Verify that Metricbeat pods are running  #

Example: 

kubectl get pods -n kube-system -o wide 
AWS Metricbeats processes

NOTE: Metricbeat will upload one pod per node to collect metrics

Check the pod logs to see if metrics collection events are being generated  #

Example: 

Metricbeats logs

Verify that after a few minutes of the pod running, files are being exported to the S3 integration bucket:  #

Example: 

AWS S3 reports list
  • <BUCKET_NAME> : name of the bucket
  • <CLUSTER_NAME_PREFIX> : cluster name, according to the prefix settings in step 3.2
  • <NODE_NAME> : name of the node where the pod that exported the metric is running

File export  #

Due to Metricbeat limitations, only 1024 log files are preserved. For the system to function properly, at least the last 7 days of log files must be preserved – we recommend, however, that they be kept for at least 35 days. 

Since the available configuration is by size and not by time, we recommend the following: 

  • Leave the default setting (which is 10 MB per file) for 1 day;
  • After exactly 24 hours, check the number of files generated: 
  • If more than 145 files were generated, please let us know as the bucket will not retain files for a week; 
  • If 29 or more files were generated, your configuration is fine;
  • If the number of files is less than 29, apply the following formula: 
FILESIZE = 10240 / 29 * QUANTITY 

For example, if 5 files were generated: 

FILESIZE = 10240 / 29 * 5 = 1765 

So inside metricbeat-deployment-template-eks-s3.yml file, set data -> metricbeat.yml: -> output.file ->  rotate_every_kb  with value of 1765 instead of 10240.