{"id":4962,"date":"2020-04-15T08:00:18","date_gmt":"2020-04-15T02:30:18","guid":{"rendered":"https:\/\/www.upnxtblog.com\/?p=4962"},"modified":"2022-02-23T17:58:35","modified_gmt":"2022-02-23T12:28:35","slug":"what-are-the-key-kubernetes-metrics-that-you-have-to-monitor","status":"publish","type":"post","link":"https:\/\/www.upnxtblog.com\/index.php\/2020\/04\/15\/what-are-the-key-kubernetes-metrics-that-you-have-to-monitor\/","title":{"rendered":"What are the key Kubernetes metrics that you have to monitor ?"},"content":{"rendered":"
<\/div>

We have already looked at BEST Kubernetes monitoring tools,<\/a> with the increasing adoption of containers and microservices<\/a> in the enterprises, monitoring utilities have to handle more services and server instances than ever before. Kubernetes environments vary from deployment to deployment, but they generally have a handful of key components, resources, and potential errors in common. Currently, the Kubernetes ecosystem provides two add-ons for aggregating and reporting monitoring data from your cluster: (1) Metrics Server and (2) kube-state-metrics.<\/strong><\/p>\n

Metrics Server<\/a>\u00a0<\/strong>is a cluster add-on that collects resource usage data from each node and provides aggregated metrics through\u00a0the Metrics API<\/a>. Metrics Server makes resource metrics such as CPU and memory available for users to query, as well as for the Kubernetes\u00a0<\/span>Horizontal Pod Autoscaler\u00a0to use for auto-scaling workloads.<\/span><\/p>\n

In addition to monitoring the CPU and memory usage of cluster nodes and pods, you will also need a way to collect metrics tracking the high-level status of the cluster and its constituent objects. Kubernetes API server exposes data about the count, health, and availability of pods, nodes, and other Kubernetes objects. By installing the kube-state-metrics<\/a> add-on in your cluster, you can consume these metrics to detect and resolve issues with cluster infrastructure, resource constraints, or pod scheduling.<\/p>\n

kube-state-metrics<\/strong>\u00a0service provides additional cluster information that Metrics Server does not. Metrics Server exposes statistics about the\u00a0resource utilization<\/em>\u00a0of Kubernetes objects, whereas kube-state-metrics listens to the Kubernetes API and generates metrics about the\u00a0state<\/em>\u00a0of Kubernetes objects: node status, node capacity (CPU and memory), number of desired\/available\/unavailable\/updated replicas per Deployment, pod status (e.g., waiting, running, ready), and so on.<\/p>\n

In this post, we are going to look at what are the key metrics and alerts that are required to monitor your Kubernetes cluster.<\/p>\n

At a high level, below are the key metrics to monitor<\/p>\n

    \n
  1. Cluster state metrics<\/li>\n
  2. Resource metrics<\/li>\n
  3. Control plane metrics<\/li>\n
  4. Kubernetes events<\/li>\n<\/ol>\n

    Key Kubernetes metrics to monitor<\/h2>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
    What to monitor?<\/strong><\/td>\n\u00a0Metrics to monitor\u00a0<\/strong><\/td>\n\u00a0Alert Criteria<\/strong><\/td>\n<\/tr>\n
    Cluster state<\/strong><\/td>\nMonitor the aggregated resources usage across all nodes in your cluster.<\/p>\n
      \n
    • Node status<\/li>\n
    • Desired pods<\/li>\n
    • Current pods<\/li>\n
    • Available pods<\/li>\n
    • Unavailable pods<\/li>\n<\/ul>\n<\/td>\n
    		\n
      \n
    • Node status<\/li>\n
    • Desired vs. current pods<\/li>\n
    • Available and unavailable pods<\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
    Node resources<\/strong><\/td>\nFor each of the node monitor :<\/p>\n
      \n
    • Memory requests<\/li>\n
    • Memory limits<\/li>\n
    • Allocatable memory<\/li>\n
    • Memory utilization<\/li>\n
    • CPU requests<\/li>\n
    • CPU limits<\/li>\n
    • Allocatable CPU<\/li>\n
    • CPU utilization<\/li>\n
    • Disk utilization<\/li>\n<\/ul>\n<\/td>\n
    		If the node\u2019s CPU or memory usage drops below a desired threshold.<\/p>\n
      \n
    • Memory limits per pod vs. memory utilization per pod<\/li>\n
    • Memory utilization<\/li>\n
    • Memory requests per node vs. allocatable memory per node<\/li>\n
    • Disk utilization<\/li>\n
    • CPU requests per node vs. allocatable CPU per node<\/li>\n
    • CPU limits per pod vs. CPU utilization per pod<\/li>\n
    • CPU utilization<\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
    Missing pod<\/strong><\/td>\nHealth and availability of your pod deployments.<\/p>\n
      \n
    • Available pods<\/li>\n
    • Unavailable pods<\/li>\n<\/ul>\n<\/td>\n
    		If the number of available pods for a deployment falls below the number of pods you specified when you created the deployment.<\/td>\n<\/tr>\n
    Pods that are not running<\/strong><\/td>\nIf a pod isn\u2019t running or even scheduled, there could be an issue with either the pod or the cluster, or with your entire Kubernetes deployment.<\/p>\n
      \n
    • Pod status<\/li>\n<\/ul>\n<\/td>\n
    		Alerts should be based on the status of your pods (\u201cFailed,\u201d \u201dPending,\u201d or \u201cUnknown\u201d for the period of time you specify)<\/td>\n<\/tr>\n
    Container restarts<\/strong><\/td>\nContainer restarts could happen when you’re hitting a memory limit (ex.Out of Memory kills) in your containers.<\/p>\n

    Also, there could be an issue with either the container itself or its host.<\/td>\n

    		Kubernetes automatically restarts containers,\u00a0 but setting up an alert will give you an immediate notification later you can analyze and set the proper limits<\/td>\n<\/tr>\n
    Container resource usage<\/strong><\/td>\nMonitor container resource usage for containers in case you’re hitting resource limits, spikes in resource consumption,<\/td>\n\u00a0Alerts to check if container CPU and memory usage and on limits are based on thresholds.<\/td>\n<\/tr>\n
    Storage volumes<\/strong><\/td>\nMonitor storage to<\/p>\n
      \n
    • Ensure your application has enough disk space so pods don\u2019t run out of space.<\/li>\n
    • Volume usage and adjust either the amount of data generated by the application or the size of the volume according to usage.<\/li>\n<\/ul>\n<\/td>\n
    		Alerts to check if available bytes, capacity crosses your thresholds.<\/p>\n

    Identify persistent volumes and apply a different alert threshold or notification for these volumes, which likely hold important application data.<\/td>\n<\/tr>\n

    Control Plane – Etcd<\/strong><\/td>\nMonitor etcd for the below parameters:<\/p>\n
      \n
    • Leader existence and change rate<\/li>\n
    • Committed, applied, pending, and failed proposals.<\/li>\n
    • gRPC performance.<\/li>\n<\/ul>\n<\/td>\n
    		Alerts to check if any pending or failed proposals or reach inappropriate thresholds.<\/td>\n<\/tr>\n
    Control Plane – API Server<\/strong><\/td>\nMonitor the API server for below parameters :<\/p>\n
      \n
    • Rate \/ number of HTTP requests<\/li>\n
    • Rate\/number of apiserver requests<\/li>\n<\/ul>\n<\/td>\n
    		Alerts to check if the rate or number of HTTP requests crosses a desired threshold.<\/td>\n<\/tr>\n
    Control Plane – Scheduler<\/strong><\/td>\nMonitor the scheduler for the below parameters<\/p>\n
      \n
    • Rate, number, and latency of HTTP requests.<\/li>\n
    • Scheduling latency.<\/li>\n
    • Scheduling attempts by result.<\/li>\n
    • End-to-end scheduling latency (sum of scheduling).<\/li>\n<\/ul>\n<\/td>\n
    		Alerts to check if the rate or number of HTTP requests crosses a desired threshold.<\/td>\n<\/tr>\n
    Control Plane – Controller Manager<\/strong><\/td>\nMonitor the scheduler for the below parameters:<\/p>\n
      \n
    • Work queue depth<\/li>\n
    • Number of retries handled by the work queue<\/li>\n<\/ul>\n<\/td>\n
    		Alerts to check if requests to the work queue exceed a maximum threshold.<\/td>\n<\/tr>\n
    Kubernetes events<\/strong><\/td>\nCollecting events from Kubernetes and from the container engine (such as Docker) allows you to see how pod creation, destruction, starting, or stopping affects the performance of your infrastructure.<\/td>\nAny failure or exception should need to be alerted.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

     <\/p>\n

    I hope, I have covered key metrics and alerts that are required to monitor your Kubernetes cluster. Also If I have missed out on any of the key metrics, do let me know.<\/p>\n

    Like this post? Don\u2019t forget to share it!<\/strong><\/em><\/p>\n

    Useful Resources<\/h2>\n