Part 8 – Adding Metricbeat to the EFK Stack on OKE

Metricbeat is a lightweight shipper that you can install on your servers to periodically collect metrics from the operating system and from services running on the server. In this guide, Metricbeat can take metrics and statistics from both K8S Master and K8S Nodes and ship them directly to Elasticsearch with recommended templates.

ElasticSearch’s Metricbeat is a lightweight shipper of both system and application metrics that runs as an agent on a client host.  That means that along with standard CPU/mem/disk/network metrics, you can also monitor Apache, Docker, Nginx, Redis, etc. as well as create your own collector in the Go language.

Metricbeat is not a replacement for monitoring systems, but instead, it helps you to see hardware and software trends and predict load, resource usage, and other stats which could be used for Business Intelligence.

For example, Metricbeat could help monitor servers by collecting metrics from these systems and services:

Web servers

  • Apache
  • Nginx

Databases

  • MongoDB
  • MySQL
  • PostgreSQL
  • Redis

Other services

  • HAProxy
  • System
  • Zookeeper

Metricbeat can insert collected metrics not only into Elasticsearch or fluentd but also Redis and Apache Kafka. In this article, we will ship data directly to Elasticsearch, but keep in mind that there are other options.

Install Metricbeat on OKE Nodes

By design, OKE provides Public IP Address on all of the nodes in a node pool when you deploy. It’s also optional (but a good practice) to supply the SSH Public Key so that we can connect to the nodes later on. Assume that it has been taken care during node pool creation, we need to access the nodes through SSH and then install metric beat on top of it.

Here you can see that I have 3 nodes in my node pool that I need to SSH into and install the Metricbeat binary. Once you SSH onto one of your nodes, just run the following command.

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-6.2.4-x86_64.rpm
sudo rpm -vi metricbeat-6.2.4-x86_64.rpm

This will install the Metricbeat on your node. Repeat this on your remaining nodes.

Configuring Metricbeat on OKE Node

To configure Metricbeat, you edit the configuration file. For rpm and deb, you’ll find the configuration file at /etc/metricbeat/metricbeat.yml

Metricbeat uses modules to collect metrics. You configure each module individually. For this article, I have already uploaded a sample config file here. You can refer this file and change according to your need. For an example, you must change the hostname of elasticsearch and kibana in your config file as those will be different for you.

My module set has configured these modules:

metricbeat.modules:
- module: system
  period: 10s
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    - core
    - diskio
    - socket
  processes: ['.*']
  process.include_top_n:
    by_cpu: 5
    by_memory: 5
- module: system
  period: 1m
  metricsets:
    - filesystem
    - fsstat
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
- module: system
  period: 15m
  metricsets:
    - uptime
- module: docker
  metricsets:
    - container
    - cpu
    - diskio
    - healthcheck
    - info
    - memory
    - network
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
  enabled: true
- module: kubernetes
  metricsets:
    - node
    - system
    - pod
    - container
    - volume
  period: 10s
  hosts: ["localhost:10255"]
  enabled: true
- module: kubernetes
  enabled: false
  metricsets:
    - state_node
    - state_deployment
    - state_replicaset
    - state_pod
    - state_container
  period: 10s
  hosts: ["kube-state-metrics:8080"]
- module: kubernetes
  enabled: false
  metricsets:
    - event

You need to update the Kibana config host according to your Kibana IP Address.

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "10.96.148.68:5601"

You also need to change the config for Elasticsearch output. Change the IP Address of elasticsearch according to your environment:

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.96.196.142:9200"]

In Elasticsearch, index templates are used to define settings and mappings that determine how fields should be analyzed.

The recommended index template file for Metricbeat is installed by the Metricbeat packages. If you accept the default configuration in the metricbeat.yml config file, Metricbeat loads the template automatically after successfully connecting to Elasticsearch. If the template already exists, it’s not overwritten unless you configure Metricbeat to do so. It will set up the index as metricbeat-*

Setup the Kibana Dashboard

Metricbeat comes packaged with example Kibana dashboards, visualizations, and searches for visualizing Metricbeat data in Kibana. Before you can use the dashboards, you need to create the index pattern metricbeat-*and load the dashboards into Kibana. To do this, you can either run the commandsetup (as described here).

sudo metricbeat setup --dashboards

Run Metricbeat by issuing the appropriate command for your platform.

sudo systemctl start metricbeat
sudo systemctl status metricbeat
sudo head -n30 /var/log/metricbeat/metricbeat

Now head over to the Kibana Dashboard and see the pre-populated Metricbeat Dashboards are already there. You can click on Dashboard and select [Metricbeat System] Host Overview.

For the Docker containers, select [Metricbeat System] Docker Overview.

 

About Prasenjit Sarkar

Prasenjit Sarkar is a Product Manager at Oracle for their Public Cloud with primary focus on Cloud Strategy, Cloud Native Applications and API Platform. His primary focus is driving Oracle’s Cloud Computing business with commercial and public sector customers; helping to shape and deliver on a strategy to build broad use of Oracle’s Infrastructure as a Service (IaaS) offerings such as Compute, Storage, Network & Database as a Service. He is also responsible for developing public/private cloud integration strategies, customer’s Cloud Computing architecture vision, future state architectures, and implementable architecture roadmaps in the context of the public, private, and hybrid cloud computing solutions Oracle can offer.

Leave a Reply