Security in Google Cloud Platform. Launch your Career in Cloud Security.
AI For Everyone: AI Training For Business Professionals
Build AI Powered Chatbots without Programming

Monitoring Docker containers using Prometheus + cAdvisor + Grafana

With the increasing adoption of containers and microservices in the enterprises, monitoring utilities have to handle more services and server instances than ever before.Support for multi-dimensional data collection, querying and perfect dashboard visualization tool is a great strength and makes right fit to be part of your operational toolset.In this post,we take look at  Monitoring Docker containers using Prometheus + cAdvisor + Grafana.

This quickstart assumes basic understanding of Docker concepts, please refer earlier posts for understanding on Docker & how to install and containerize applications.

Architecture

We are going to use following tools to collect, aggregate & visualize metrics.

  1. Container metrics using cAdvisor
  2. Host metrics using Prometheus Node Exporter
  3. Alerting using Prometheus Alert Manager
  4. Visualization using Grafana pre-built dashboards

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloudPrometheus collects metrics from monitored targets by scraping metrics HTTP endpoints,we are going to configure it to scrape metrics for containers from cAdvisor and node metrics from Prometheus Node exporter.

nordvpn 3Y deal
Monitoring Architecture

Image – Monitoring Architecture

#1.Prepare Docker Compose Configuration

Create new Project directory and prepare Docker compose file like the one below

Subscribe to Emails

More than 1000+ subscribers across the world trust & read Upnxtblog articles. Subscribe to emails updates!
* indicates required
version: '3'
services:
prometheus:
    image: 'prom/prometheus:v2.1.0'
    container_name: prometheus
    ports:
    - '9090:9090'
    command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--storage.tsdb.path=/prometheus'
    - '--web.console.libraries=/usr/share/prometheus/console_libraries'
    - '--web.console.templates=/usr/share/prometheus/consoles'
    volumes:
    - './prometheus.yml:/etc/prometheus/prometheus.yml:ro'
    - './alert.rules:/etc/prometheus/alert.rules'
    depends_on:
    - cadvisor
    networks:
    - samplenet
cadvisor:
    image: google/cadvisor
    container_name: cadvisor
    ports:
    - '8081:8081'
    volumes:
    - '/:/rootfs:ro'
    - '/var/run:/var/run:rw'
    - '/sys:/sys:ro'
    - '/var/lib/docker/:/var/lib/docker:ro'
    networks:
    - samplenet
alertmanager:
    image: prom/alertmanager
    ports:
    - '9093:9093'
    volumes:
    - './alertmanager/:/etc/alertmanager/'
    restart: always
    command:
    - '--config.file=/etc/alertmanager/config.yml'
    - '--storage.path=/alertmanager'
    networks:
    - samplenet
node-exporter:
    image: prom/node-exporter
    volumes:
     - '/proc:/host/proc:ro'
     - '/sys:/host/sys:ro'
     - '/:/rootfs:ro'
    command:
     - '--path.procfs=/host/proc'
     - '--path.sysfs=/host/sys'
     - '--collector.filesystem.ignored-mount-points'
     - ^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)
    ports:
     - '9100:9100'
    networks:
     - samplenet
    restart: always
grafana:
    image: grafana/grafana
    depends_on:
    - prometheus
    ports:
    - '3000:3000'
    volumes:
    - 'grafana_data:/var/lib/grafana'
    - './grafana/provisioning/:/etc/grafana/provisioning/'
    env_file:
    - ./grafana/config.monitoring
    networks:
    - samplenet
    restart: always
volumes:
grafana_data: {}
prometheus_data: {}

#2.Prometheus Node Exporter Configuration

On the compose file,mount the host /proc and /sys directory so that the container has accessed to the necessary information to report on.

#3.cAdvisor Configuration

On the compose file,mount the var/run, /sys directory /var/lib/docker so that the cAdvisor can collect container metrics and report to Prometheus.

#4.Prometheus Configuration

Create new Prometheus configuration in a file called prometheus.yml . Prometheus server requires a configuration file that defines the endpoints to scrape along with how frequently the metrics should be accessed and to define the servers and ports that Prometheus should scrape data from. In the below example, we have defined 3 targets (1st one for Prometheus itself , 2nd One for collecting container metrics using cAdvisor and last one is for node metrics using Node exporter) running on different ports.

AI For Everyone: AI Training For Business Professionals
scrape_configs:
- job_name: prometheus
scrape_interval: 5s
static_configs:
- targets:
- prometheus:9090

- job_name: cadvisor
scrape_interval: 5s
static_configs:
- targets:
- cadvisor:8081

- job_name: node-exporter
scrape_interval: 5s
static_configs:
- targets:
- node-exporter:9100

alerting:
alertmanagers:
- static_configs:
- targets: 
- alertmanager:9093

rule_files:
- 'alert.rules'

For a complete specification of configuration options, see the configuration documentation.

#5.Alert Manager Configuration

Alert Manager takes care of de-duplicating, grouping, and routing the alerts to the correct receiver integration such as email, slack channels etc.,

We are going to make use of default configuration like example below,create new folder alertmanager and paste the configuration file (config.yml)

route:
   receiver: 'slack'

receivers:
   - name: 'slack'
      slack_configs:
      - send_resolved: true
      username: '<username>'
      channel: '#<channel-name>'
      api_url: '<incomming-webhook-url>'

#6.Grafana Configuration

Grafana would be the dashboard visualization tool of choice for Prometheus users and support for Grafana ships with the tool.For monitoring Docker containers,we are going to import pre-built dashboards from Grafana.com.

Create new folder Grafana and copy the contents from the source to automate provisioning of datasources & dashboards.For the full Grafana installation instructions, see the official Grafana documentation.

nordvpn

Finally our folder would look like the one below.

Monitoring Folder structure

Image – Monitoring Folder structure

Congrats! we have now configured all tools to monitor our containers,lets start the compose and check.

#7.View Metrics

Start the Docker compose using docker-compose up -d

Once all the containers are up, Prometheus will now scrape and store the data based on the configuration. Prometheus is configured on port 9090,Go to the dashboard http://localhost:9090 and verify that Prometheus now has information about the time series information on the containers,node.

Use the dropdown next to the “Execute” button to see a list of metrics this server is collecting. In the list you’ll see a number of metrics prefixed with node_, that have been collected by the Node Exporter. For example, you can see the node’s CPU usage via the node_cpu metric.

With this expression browser,you can enter any expression and see its result either in a table or graphed over time.

Prometheus Dashboard

Image – Prometheus Dashboard

Grafana will be listening on http://localhost:3000. The default login is "admin" / "admin",once you login use “Filter” option to browse for dashboards.From the list,choose “Docker Prometheus Monitoring” to view it.

Container Metrics / Alerts

Image – Container Metrics / Alerts

If you want to modify Dashboard or datasource,you should manually edit the downloaded JSON files and correct the datasource: entries to reflect the Grafana data source name which you chose for your Prometheus server. Use the “Dashboards” → “Home” → “Import” option to import the edited dashboard file into your Grafana install.

All source files used for this article can be found here.

In this post,we have got introduced on how to setup targets in Prometheus, configure Prometheus, cAdvisor, Node Exporter and Alert Manager to monitor both containers and host resources.

Like this post? Don’t forget to share it!

Deep Learning Specialization on Coursera

Additional Resources

 

Summary
Monitoring Docker containers using Prometheus + cAdvisor + Grafana
Article Name
Monitoring Docker containers using Prometheus + cAdvisor + Grafana
Description
In this post,we take look at how to monitoring Docker containers using Prometheus + cAdvisor + Grafana.
Author
Publisher Name
upnxtblog
Publisher Logo
%d bloggers like this: