{"id":3278,"date":"2019-01-10T07:00:53","date_gmt":"2019-01-10T01:30:53","guid":{"rendered":"http:\/\/www.upnxtblog.com\/?p=3278"},"modified":"2020-12-21T19:04:21","modified_gmt":"2020-12-21T13:34:21","slug":"how-to-aggregate-docker-container-logs-and-analyse-with-elk-stack","status":"publish","type":"post","link":"https:\/\/www.upnxtblog.com\/index.php\/2019\/01\/10\/how-to-aggregate-docker-container-logs-and-analyse-with-elk-stack\/","title":{"rendered":"How to aggregate Docker Container logs and analyse with ELK stack ?"},"content":{"rendered":"
<\/div>

Today we are going to learn about how to aggregate Docker container logs and analyze the same centrally using ELK stack<\/a>.<\/strong>\u00a0ELK<\/strong>\u00a0stack comprises of Elasticsearch, Logstash, and Kibana <\/strong>tools. Elasticsearch<\/a> <\/strong>is a highly scalable open-source full-text search and analytics engine.<\/p>\n

It allows you to store, search, and analyze big volumes of data quickly and in near real-time. Kibana<\/a> <\/strong>is like a window into the Elastic Stack. It enables visual exploration and real-time analysis of your data in Elasticsearch. Logstash<\/a> <\/strong>is the central dataflow engine in the Elastic Stack for gathering, enriching, and unifying all of your data regardless of format or schema. If you want to learn more about key concepts of the ELK stack, please check out earlier posts here<\/a>.<\/p>\n

As your container volume increases, it’s difficult to manage them and their logs. There is a need for a centralized solution to take care of log aggregation, monitoring, and analysis. Luckily we already have ELK stack which does Log aggregation well but Docker container logs need to be routed to Logstash.<\/p>\n

For log routing from each of the containers, we are going to use Logspout<\/a>\u00a0<\/strong>utility that attaches to all containers on a host, then routes their logs wherever we want. Here in our case, we are going to push it to Logstash and let it handle shipping, transformation, etc., In this article, we are going to use ElasticSearch<\/strong> to store, index the logs. Logstash<\/strong> ships manage to transform logs into a consistent format and use Kibana<\/strong> to visualize the logs.<\/p>\n

This quickstart assumes a basic understanding of Docker concepts<\/a>, please refer to earlier posts for understanding Docker & how to install<\/a> and containerize applications<\/a>.<\/p>\n

With this context now, let’s check out how to aggregate Docker Container logs and analyze the same.<\/p>\n\n

Log Aggregation Architecture<\/h2>\n

Before we head to the tutorial, below is what we want to achieve. All logs from the Docker containers will be routed to Logstash using Logspout over UDP protocol. <\/em>Logstash will then serve as a Data collection engine, pushes it to Elasticsearch for indexing, making it available for searching. Post which using Kibana, we can analyze the logs, create visualizations as we want.<\/p>\n

\"Docker
Image – Docker Log aggregation using Logspout,ELK stack<\/figcaption><\/figure>\n

Next, we head over to the implementation of the same, here is an overview of the steps involved<\/p>\n

    \n
  1. Prepare Docker Compose scripts for ELK stack and Logspout configuration<\/li>\n
  2. Launch Docker Containers<\/li>\n
  3. Define an index pattern<\/li>\n
  4. Visualize the data<\/li>\n<\/ol>\n

    Step #1. Prepare Docker Compose scripts for ELK stack and Logspout configuration<\/h2>\n

    ElasticSearch Docker Configuration:<\/strong>\u00a0We are going to use the official image, expose the two ports (9200\/9300) as required. In production environments, make sure that the above ports are only accessible from internal and restrict access to the public.<\/em><\/p>\n

    elasticsearch:\r\nimage: elasticsearch:1.5.2\r\nports:\r\n- '9200:9200'\r\n- '9300:9300'\r\n<\/pre>\n
     <\/p>\n
    Kibana Docker Configuration: <\/strong>Kibana needs to connect to an instance of ElasticSearch so that visualizations can be made. Add ELASTICSEARCH_URL<\/code>\u00a0environment variable and specify ElasticSearch Instance to connect to. #5601<\/em> default port needs to be exposed.<\/p>\n
    kibana:\r\nimage: kibana:4.1.2\r\nlinks:\r\n- elasticsearch\r\nenvironment:\r\n- ELASTICSEARCH_URL=http:\/\/elasticsearch:9200\r\nports:\r\n- '5601:5601'\r\ndepends_on:\r\n- elasticsearch\r\n\r\n<\/pre>\n
    Logstash Docker Configuration: <\/strong>Logstash can process data from any source and normalizes it for storing. In the command section, you can note that Logstash will receive input on UDP protocol at port #5000<\/em> and pushes the data to ElasticSearch instance.<\/p>\n
    Also note, below is for demonstration purpose only but actually, Logstash can dynamically unify data from various sources and normalize the data into any of the destinations. You can also cleanse your data for diverse advanced downstream analytics and visualization use cases.<\/p>\n
    logstash:\r\n\u00a0\u00a0\u00a0 image: logstash:2.1.1\r\n\u00a0\u00a0\u00a0 environment:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 - STDOUT=true\r\n\u00a0\u00a0\u00a0 links:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 - elasticsearch\r\n\u00a0\u00a0\u00a0 depends_on:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - elasticsearch\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - kibana\r\n\u00a0\u00a0\u00a0 command: 'logstash -e \"input { udp { port => 5000 } } output { elasticsearch { hosts => elasticsearch } }\"'\r\n\r\n<\/pre>\n
    We now have an ELK stack configuration ready. Next steps, we’ll explore how to push logs into the system using Logspout.<\/p>\n
    Logspout Docker Configuration: <\/strong>Logspout will monitor Docker events. When a new container is launched it will automatically start collecting its logs. Every log line will be pushed into Logstash using the UDP protocol. Below is the Docker configuration, we are using logspout v3 but there are the latest versions available.<\/p>\n
    \u00a0 logspout:\r\n\u00a0\u00a0\u00a0 image: gliderlabs\/logspout:v3\r\n\u00a0\u00a0\u00a0 command: 'udp:\/\/logstash:5000'\r\n\u00a0\u00a0\u00a0 links:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 - logstash\r\n\u00a0\u00a0\u00a0 volumes:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 - '\/var\/run\/docker.sock:\/tmp\/docker.sock'\r\n\u00a0\u00a0\u00a0 depends_on:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - elasticsearch\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - logstash\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - kibana\r\n<\/pre>\n
     <\/p>\n
    Finally, our Docker Compose configuration will look the one below<\/p>\n
    version: '3.3'\r\nservices:\r\nlogspout:\r\nimage: gliderlabs\/logspout:v3\r\ncommand: 'udp:\/\/logstash:5000'\r\nlinks:\r\n- logstash\r\nvolumes:\r\n- '\/var\/run\/docker.sock:\/tmp\/docker.sock'\r\ndepends_on:\r\n- elasticsearch\r\n- logstash\r\n- kibana\r\nlogstash:\r\nimage: logstash:2.1.1\r\nenvironment:\r\n- STDOUT=true\r\nlinks:\r\n- elasticsearch\r\ndepends_on:\r\n- elasticsearch\r\n- kibana\r\ncommand: 'logstash -e \"input { udp { port => 5000 } } output { elasticsearch { hosts => elasticsearch } }\"'\r\nkibana:\r\nimage: kibana:4.1.2\r\nlinks:\r\n- elasticsearch\r\nenvironment:\r\n- ELASTICSEARCH_URL=http:\/\/elasticsearch:9200\r\nports:\r\n- '5601:5601'\r\ndepends_on:\r\n- elasticsearch\r\nelasticsearch:\r\nimage: elasticsearch:1.5.2\r\nports:\r\n- '9200:9200'\r\n- '9300:9300'\r\n<\/pre>\n
    Step #2.Launch Docker Containers<\/h2>\n
    Now that Docker compose script is ready, launch containers using docker-compose up<\/code> command.<\/p>\n
    \"Start
    Image – Start ELK stack<\/figcaption><\/figure>\n
    ElasticSearch and Kibana can take a few minutes to start. When Logstash launches it starts generating indexes in Elasticsearch.If you have noticed, we have not created any application which is generating logs, here we are going to use startup logs generated by ElasticSearch, Kibana & Logstash itself.<\/p>\n
    If you want to ignore logs for a specific container then you can add LOGSPOUT=ignore<\/code> as an environment variable on Docker compose a script. For more information on other Logspout environment variables, please check here<\/a>.<\/p>\n
    Once all the containers are up, the next step is to launch Kibana and start defining the index pattern.<\/p>\n
    Step #3.Define Index Pattern<\/h2>\n
    Now that startup logs would have been loaded to Elasticsearch, we would need to create an index pattern. An index is a collection of documents that have similar characteristics. An index is identified by a name and this name is used to refer to the index when performing indexing, search, update, and delete operations against the documents in it. Indexing is similar to the creation and update process of CRUD operations.<\/p>\n
    Launch Kibana on port # 5601<\/em>,under ‘Indices’ \/ ‘Management’ (on latest versions) tab you can find option to create Index pattern.Enter the name of the index ex.logstash-*<\/em>. Kibana will then ask for a field containing a time\/timestamp which it should use for visualizing time-series data. for our case, this is the “@timestamp” field.<\/p>\n
    \"Define
    Image – Define Index Pattern<\/figcaption><\/figure>\n
    Now that we have created the index pattern, it would take a few minutes to complete. The next step is to create visualizations. Before that, we can check the data from the ‘Discover’ tab.<\/p>\n
    \"Check
    Image – Check the log data under Discover tab<\/figcaption><\/figure>\n
    We have enough data to visualize, we are ready to create Visualization<\/p>\n
    Step #4.Visualize the data<\/h2>\n
    For demonstration purposes, I have created the below visualizations and attached them to the Dashboard.<\/p>\n
      \n
    1. Metrics chart to display count of log events<\/em><\/li>\n
    2. Area chart to show\u00a0count of logs events against time<\/em><\/li>\n<\/ol>\n
      \"Log
      Image – Log events Dashboard with visualizations<\/figcaption><\/figure>\n
      Congrats! we have learned how to aggregate all Docker container logs and analyze the same centrally using ELK stack.<\/em><\/p>\n
      As always there is more to what was covered here! If you have questions, please post it in the comments section.<\/p>\n
      Like this post? Don\u2019t forget to share it!<\/strong><\/em><\/p>\n
      Useful Resources<\/h2>\n