Metricbeat is a lightweight shipper that you can install on your servers to periodically collect metrics from the operating system and from services running on the server. In this guide, Metricbeat can take metrics and statistics from both K8S Master and K8S Nodes and ship them directly to Elasticsearch with recommended templates.
ElasticSearch’s Metricbeat is a lightweight shipper of both system and application metrics that runs as an agent on a client host. That means that along with standard CPU/mem/disk/network metrics, you can also monitor Apache, Docker, Nginx, Redis, etc. as well as create your own collector in the Go language.
Metricbeat is not a replacement for monitoring systems, but instead, it helps you to see hardware and software trends and predict load, resource usage, and other stats which could be used for Business Intelligence.
For example, Metricbeat could help monitor servers by collecting metrics from these systems and services:
Metricbeat can insert collected metrics not only into Elasticsearch or fluentd but also Redis and Apache Kafka. In this article, we will ship data directly to Elasticsearch, but keep in mind that there are other options.
Install Metricbeat on OKE Nodes
By design, OKE provides Public IP Address on all of the nodes in a node pool when you deploy. It’s also optional (but a good practice) to supply the SSH Public Key so that we can connect to the nodes later on. Assume that it has been taken care during node pool creation, we need to access the nodes through SSH and then install metric beat on top of it.
Here you can see that I have 3 nodes in my node pool that I need to SSH into and install the Metricbeat binary. Once you SSH onto one of your nodes, just run the following command.
curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-6.2.4-x86_64.rpm sudo rpm -vi metricbeat-6.2.4-x86_64.rpm
This will install the Metricbeat on your node. Repeat this on your remaining nodes.
Configuring Metricbeat on OKE Node
To configure Metricbeat, you edit the configuration file. For rpm and deb, you’ll find the configuration file at
Metricbeat uses modules to collect metrics. You configure each module individually. For this article, I have already uploaded a sample config file here. You can refer this file and change according to your need. For an example, you must change the hostname of elasticsearch and kibana in your config file as those will be different for you.
My module set has configured these modules:
metricbeat.modules: - module: system period: 10s metricsets: - cpu - load - memory - network - process - process_summary - core - diskio - socket processes: ['.*'] process.include_top_n: by_cpu: 5 by_memory: 5 - module: system period: 1m metricsets: - filesystem - fsstat processors: - drop_event.when.regexp: system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)' - module: system period: 15m metricsets: - uptime - module: docker metricsets: - container - cpu - diskio - healthcheck - info - memory - network hosts: ["unix:///var/run/docker.sock"] period: 10s enabled: true - module: kubernetes metricsets: - node - system - pod - container - volume period: 10s hosts: ["localhost:10255"] enabled: true - module: kubernetes enabled: false metricsets: - state_node - state_deployment - state_replicaset - state_pod - state_container period: 10s hosts: ["kube-state-metrics:8080"] - module: kubernetes enabled: false metricsets: - event
You need to update the Kibana config host according to your Kibana IP Address.
#============================== Kibana ===================================== # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API. # This requires a Kibana endpoint configuration. setup.kibana: # Kibana Host # Scheme and port can be left out and will be set to the default (http and 5601) # In case you specify and additional path, the scheme is required: http://localhost:5601/path # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601 host: "10.96.148.68:5601"
You also need to change the config for Elasticsearch output. Change the IP Address of elasticsearch according to your environment:
#-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["10.96.196.142:9200"]
In Elasticsearch, index templates are used to define settings and mappings that determine how fields should be analyzed.
The recommended index template file for Metricbeat is installed by the Metricbeat packages. If you accept the default configuration in the
metricbeat.yml config file, Metricbeat loads the template automatically after successfully connecting to Elasticsearch. If the template already exists, it’s not overwritten unless you configure Metricbeat to do so. It will set up the index as metricbeat-*
Setup the Kibana Dashboard
Metricbeat comes packaged with example Kibana dashboards, visualizations, and searches for visualizing Metricbeat data in Kibana. Before you can use the dashboards, you need to create the index pattern
metricbeat-*and load the dashboards into Kibana. To do this, you can either run the command
setup (as described here).
sudo metricbeat setup --dashboards
Run Metricbeat by issuing the appropriate command for your platform.
sudo systemctl start metricbeat sudo systemctl status metricbeat sudo head -n30 /var/log/metricbeat/metricbeat
Now head over to the Kibana Dashboard and see the pre-populated Metricbeat Dashboards are already there. You can click on Dashboard and select [Metricbeat System] Host Overview.
For the Docker containers, select [Metricbeat System] Docker Overview.