{"id":3594,"date":"2019-03-05T07:00:13","date_gmt":"2019-03-05T01:30:13","guid":{"rendered":"http:\/\/www.upnxtblog.com\/?p=3594"},"modified":"2020-04-23T12:34:56","modified_gmt":"2020-04-23T07:04:56","slug":"monitoring-docker-containers-using-prometheus-cadvisor-grafana","status":"publish","type":"post","link":"https:\/\/www.upnxtblog.com\/index.php\/2019\/03\/05\/monitoring-docker-containers-using-prometheus-cadvisor-grafana\/","title":{"rendered":"Monitoring Docker containers using Prometheus + cAdvisor + Grafana"},"content":{"rendered":"
<\/div>

With the increasing adoption of containers and microservices<\/a> in the enterprises, monitoring utilities have to handle more services and server instances than ever before. Support for multi-dimensional data collection, querying and perfect dashboard visualization tool is a great strength and makes the right fit to be part of your operational toolset. In this post, we take look at\u00a0\u00a0Monitoring Docker containers using Prometheus<\/a> + cAdvisor<\/a> + Grafana.<\/strong>\"\"<\/p>\n

This quickstart assumes basic understanding of Docker\u00a0concepts<\/a>, please refer to earlier posts for understanding on Docker & how to install<\/a> and containerize applications<\/a>.<\/p>\n\n

Architecture<\/h2>\n

We are going to use the following tools to collect, aggregate & visualize metrics.<\/p>\n

    \n
  1. Container metrics using cAdvisor<\/em><\/a><\/strong><\/li>\n
  2. Host metrics using Prometheus Node Exporter<\/em><\/strong><\/li>\n
  3. Alerting using Prometheus Alert Manager<\/em><\/strong><\/li>\n
  4. Visualization using Grafana<\/em><\/strong> pre-built dashboards<\/li>\n<\/ol>\n

    Prometheus<\/a>\u00a0<\/strong>is an open-source systems monitoring and alerting toolkit originally built at\u00a0SoundCloud<\/a>.\u00a0Prometheus<\/a> collects metrics from monitored targets by scraping metrics HTTP endpoints, we are going to configure it to scrape metrics for containers from cAdvisor<\/a>\u00a0and node metrics from Prometheus Node exporter.<\/p>\n

    \"Monitoring
    Image – Monitoring Architecture<\/figcaption><\/figure>\n

    #1. Prepare Docker Compose Configuration<\/h2>\n

    Create New Project directory and prepare Docker compose file like the one below<\/p>\n

    version: '3'\r\nservices:\r\nprometheus:\r\n    image: 'prom\/prometheus:v2.1.0'\r\n    container_name: prometheus\r\n    ports:\r\n    - '9090:9090'\r\n    command:\r\n    - '--config.file=\/etc\/prometheus\/prometheus.yml'\r\n    - '--storage.tsdb.path=\/prometheus'\r\n    - '--web.console.libraries=\/usr\/share\/prometheus\/console_libraries'\r\n    - '--web.console.templates=\/usr\/share\/prometheus\/consoles'\r\n    volumes:\r\n    - '.\/prometheus.yml:\/etc\/prometheus\/prometheus.yml:ro'\r\n    - '.\/alert.rules:\/etc\/prometheus\/alert.rules'\r\n    depends_on:\r\n    - cadvisor\r\n    networks:\r\n    - samplenet\r\ncadvisor:\r\n    image: google\/cadvisor\r\n    container_name: cadvisor\r\n    ports:\r\n    - '8081:8081'\r\n    volumes:\r\n    - '\/:\/rootfs:ro'\r\n    - '\/var\/run:\/var\/run:rw'\r\n    - '\/sys:\/sys:ro'\r\n    - '\/var\/lib\/docker\/:\/var\/lib\/docker:ro'\r\n    networks:\r\n    - samplenet\r\nalertmanager:\r\n    image: prom\/alertmanager\r\n    ports:\r\n    - '9093:9093'\r\n    volumes:\r\n    - '.\/alertmanager\/:\/etc\/alertmanager\/'\r\n    restart: always\r\n    command:\r\n    - '--config.file=\/etc\/alertmanager\/config.yml'\r\n    - '--storage.path=\/alertmanager'\r\n    networks:\r\n    - samplenet\r\nnode-exporter:\r\n    image: prom\/node-exporter\r\n    volumes:\r\n     - '\/proc:\/host\/proc:ro'\r\n     - '\/sys:\/host\/sys:ro'\r\n     - '\/:\/rootfs:ro'\r\n    command:\r\n     - '--path.procfs=\/host\/proc'\r\n     - '--path.sysfs=\/host\/sys'\r\n     - '--collector.filesystem.ignored-mount-points'\r\n     - ^\/(sys|proc|dev|host|etc|rootfs\/var\/lib\/docker\/containers|rootfs\/var\/lib\/docker\/overlay2|rootfs\/run\/docker\/netns|rootfs\/var\/lib\/docker\/aufs)($$|\/)\r\n    ports:\r\n     - '9100:9100'\r\n    networks:\r\n     - samplenet\r\n    restart: always\r\ngrafana:\r\n    image: grafana\/grafana\r\n    depends_on:\r\n    - prometheus\r\n    ports:\r\n    - '3000:3000'\r\n    volumes:\r\n    - 'grafana_data:\/var\/lib\/grafana'\r\n    - '.\/grafana\/provisioning\/:\/etc\/grafana\/provisioning\/'\r\n    env_file:\r\n    - .\/grafana\/config.monitoring\r\n    networks:\r\n    - samplenet\r\n    restart: always\r\nvolumes:\r\ngrafana_data: {}\r\nprometheus_data: {}\r\n\r\n<\/pre>\n
    #2.Prometheus Node Exporter Configuration<\/h2>\n
    On the compose file, mount the host \/proc<\/code> and \/sys<\/code> directory so that the container has access to the necessary information to report on.<\/p>\n
    #3.cAdvisor Configuration<\/h2>\n
    On the compose file, mount the var\/run, \/sys<\/code> directory \/var\/lib\/docker<\/code> so that the cAdvisor can collect container metrics and report to Prometheus.<\/p>\n
    #4.Prometheus Configuration<\/h2>\n
    Create new Prometheus configuration in a file called\u00a0prometheus.yml<\/code>\u00a0. Prometheus server requires a configuration file that defines the endpoints to scrape along with how frequently the metrics should be accessed and to define the servers and ports that Prometheus should scrape data from. In the below example, we have defined 3 targets (1st one for Prometheus itself, 2nd One for collecting container metrics using cAdvisor and last one is for node metrics using Node exporter<\/em>) running on different ports.<\/p>\n
    scrape_configs:\r\n- job_name: prometheus\r\nscrape_interval: 5s\r\nstatic_configs:\r\n- targets:\r\n- prometheus:9090\r\n\r\n- job_name: cadvisor\r\nscrape_interval: 5s\r\nstatic_configs:\r\n- targets:\r\n- cadvisor:8081\r\n\r\n- job_name: node-exporter\r\nscrape_interval: 5s\r\nstatic_configs:\r\n- targets:\r\n- node-exporter:9100\r\n\r\nalerting:\r\nalertmanagers:\r\n- static_configs:\r\n- targets: \r\n- alertmanager:9093\r\n\r\nrule_files:\r\n- 'alert.rules'<\/pre>\n
    For a complete specification of configuration options, see the\u00a0configuration documentation<\/a>.<\/p>\n
    #5.Alert Manager Configuration<\/h2>\n
    Alert Manager takes care of de-duplicating, grouping, and routing the alerts to the correct receiver integration such as email, slack channels, etc.,<\/p>\n
    We are going to make use of default configuration like the example below, create a new folder alertmanager<\/code> and paste the configuration file (config.yml<\/code>)<\/p>\n
    route:\r\n   receiver: 'slack'\r\n\r\nreceivers:\r\n   - name: 'slack'\r\n      slack_configs:\r\n      - send_resolved: true\r\n      username: '<username>'\r\n      channel: '#<channel-name>'\r\n      api_url: '<incomming-webhook-url>'\r\n\r\n<\/pre>\n
    #6.Grafana Configuration<\/h2>\n
    Grafana<\/a> would be the dashboard visualization tool of choice for Prometheus users and support for Grafana ships with the tool. For monitoring Docker containers, we are going to import pre-built dashboards from Grafana.com.<\/a><\/p>\n
    Create new folder Grafana<\/code> and copy the contents from the source<\/a> to automate the provisioning of data sources & dashboards. For the full Grafana installation instructions, see the official Grafana documentation<\/a>.<\/p>\n
    Finally, our folder would look like the one below.<\/p>\n
    \"Monitoring
    Image – Monitoring Folder structure<\/figcaption><\/figure>\n
    Congrats! we have now configured all tools to monitor our containers, let’s start the compose and check.<\/em><\/p>\n
    #7.View Metrics<\/h2>\n
    Start the Docker compose using docker-compose up -d<\/code><\/p>\n
    Once all the containers are up, Prometheus will now scrape and store the data based on the configuration. Prometheus is configured on port 9090<\/code>, Go to the dashboard http:\/\/localhost:9090<\/code> and verify that Prometheus now has information about the time series information on the containers, node.<\/p>\n
    Use the dropdown next to the “Execute” button to see a list of metrics this server is collecting. In the list, you’ll see a number of metrics prefixed with node_<\/code>, that have been collected by the Node Exporter. For example, you can see the node’s CPU usage via the\u00a0node_cpu<\/code>\u00a0metric.<\/p>\n
    With this expression browser, you can enter any expression and see its result either in a table or graph over time.<\/p>\n
    \"Prometheus
    Image – Prometheus Dashboard<\/figcaption><\/figure>\n
    Grafana will be listening on\u00a0http:\/\/localhost:3000<\/code>. The default login is \"admin\" \/ \"admin\"<\/code>, once you log in use the “Filter” option to browse for dashboards. From the list, choose “Docker Prometheus Monitoring<\/strong><\/em>” to view it.<\/p>\n
    \"Node
    Image – Host metrics<\/figcaption><\/figure>\n
    \"Container
    Image – Container Metrics \/ Alerts<\/figcaption><\/figure>\n
    If you want to modify Dashboard or data source, you should manually edit the downloaded JSON files and correct the datasource:<\/code>\u00a0entries to reflect the Grafana data source name which you chose for your Prometheus server. Use the “Dashboards” \u2192 “Home” \u2192 “Import” option to import the edited dashboard file into your Grafana install.<\/p>\n
    All source files used for this article can be found here<\/a>.<\/p>\n
    In this post, we have got introduced on how to set up targets in Prometheus, configure Prometheus, cAdvisor, Node Exporter and Alert Manager to monitor both containers and host resources.<\/p>\n
    Like this post? Don\u2019t forget to share it!<\/strong><\/em><\/p>\n
    Additional Resources<\/h2>\n