Metrics & Logging

Metrics Integration with Prometheus and Grafana¶

1. Deploy Prometheus¶

Prometheus scrapes metrics from Kubernetes components (e.g., kubelet, kube-apiserver) and stores them for querying. Use Helm to deploy Prometheus:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/prometheus --set server.service.type=ClusterIP

This installs Prometheus with a ClusterIP service, allowing it to scrape metrics from nodes, pods, and critical components.

2. Configure Prometheus Scrape Targets¶

Edit the Prometheus config file (prometheus.yml) to define scrape targets. For example:

scrape_configs:
  - job_name: "kubelet"
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - source_labels: [__meta_kubernetes_node_name]
        target_label: __metrics_node_name

  - job_name: "kube-apiserver"
    kubernetes_sd_configs:
      - role: endpoints
    relabel_configs:
      - source_labels: [__meta_kubernetes_endpoint_port_name]
        regex: "https"
        target_label: __metrics_scrape_port
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_endpoint_name]
        target_label: __metrics_scrape_target

  - job_name: "kube-proxy"
    kubernetes_sd_configs:
      - role: endpoints
    relabel_configs:
      - source_labels: [__meta_kubernetes_endpoint_port_name]
        regex: "https"
        target_label: __metrics_scrape_port
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_endpoint_name]
        target_label: __metrics_scrape_target

  - job_name: "pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        regex: "true"
        target_label: __metrics_scrape
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
        regex: "https?"
        target_label: __metrics_scheme
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        target_label: __metrics_path
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        target_label: __metrics_port

This configures Prometheus to scrape metrics from kubelet, kube-apiserver, kube-proxy, and pods with proper relabeling rules.

3. Set Up Grafana for Visualization¶

Deploy Grafana to visualize Prometheus metrics:

helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana --set persistence.enabled=false

After deployment, access Grafana via its service URL (e.g., http://grafana-svc.namespace:3000). Note that the namespace context depends on your deployment configuration. Add Prometheus as a data source and import pre-built dashboards using the dashboard ID 1067 (Kubernetes dashboard) via the import feature.

Logging Integration with ELK Stack¶

1. Deploy Elasticsearch, Logstash, and Kibana¶

Use Helm to deploy the ELK stack components:

# Elasticsearch
helm repo add elastic https://helm.elastic.co
helm install elasticsearch elastic/elasticsearch --set master.persistence.enabled=false

# Kibana
helm install kibana elastic/kibana --set elasticsearch.hosts=http://elasticsearch-svc:9200

# Logstash (optional)
helm install logstash elastic/logstash --set configMaps=logging.conf

This setup stores logs in Elasticsearch and provides Kibana for querying and visualization.

2. Configure Logstash for Kubernetes Logs¶

Create a Logstash configuration file (logging.conf) to collect and process logs:

input {
  beats {
    port => 5044
  }
  kubernetes {
    host => "http://kube-api-svc:443"
    port => 443
    ssl_certificate_file => "/etc/ssl/certs/ca-certificates.crt"
    ssl_verify_mode => "peer"
  }
}
output {
  elasticsearch {
    hosts => ["http://elasticsearch-svc:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}

This configures Logstash to receive logs from Kubernetes (via the Kubernetes logging plugin) and index them in Elasticsearch. Ensure the kubernetes plugin version matches your Logstash version. Install the Kubernetes plugin for Logstash using:

logstash-plugin install logstash-input-kubernetes

3. Visualize Logs in Kibana¶

Access Kibana via its service URL (e.g., http://kibana-svc.namespace:5601). Create an index pattern (e.g., logs-*) and use the Discover tab to query logs. Use the Kibana dashboard to build custom visualizations.

Security and Best Practices¶

RBAC: Ensure Prometheus and Elasticsearch have appropriate permissions to access cluster resources.
Network Policies: Restrict access to Prometheus, Elasticsearch, and Logstash to trusted services.
Structured Logging: Use tools like Fluentd as a sidecar container to parse and enrich logs before sending them to Logstash.

Key takeaways¶

Prometheus + Grafana provide real-time metrics for Kubernetes performance monitoring.
ELK stack enables centralized, searchable logging for troubleshooting and analysis.
Secure access controls and network policies are critical for observability tools.
Combine metrics and logs for a holistic view of cluster health and application behavior.