Part 5 — Monitoring

Flow

K8s cluster + apps  ---exports metrics--->  Prometheus  ---queries--->  Grafana dashboards
                                                 |
                                                 └── AlertManager --> Slack/Email

Install Prometheus (on the monitor VM)

cd /opt
sudo wget https://github.com/prometheus/prometheus/releases/download/v2.52.0/prometheus-2.52.0.linux-amd64.tar.gz
sudo tar -xzf prometheus-2.52.0.linux-amd64.tar.gz
sudo mv prometheus-2.52.0.linux-amd64 prometheus
cd prometheus
./prometheus &          # starts on port 9090

Open http://<monitor-ip>:9090 — you should see the Prometheus UI.

Configure Prometheus (prometheus.yml)

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'jenkins'
    metrics_path: /prometheus/
    static_configs:
      - targets: ['<jenkins-ip>:8080']

  - job_name: 'node-exporter'
    static_configs:
      - targets:
          - '<k8s-master-ip>:9100'
          - '<worker-1-ip>:9100'
          - '<worker-2-ip>:9100'

  - job_name: 'blackbox'
    metrics_path: /probe
    params: { module: [http_2xx] }
    static_configs:
      - targets:
          - http://<jenkins-ip>:8080
          - https://your-app.com
    relabel_configs:
      - source_labels: [__address__]
        target_label:  __param_target
      - source_labels: [__param_target]
        target_label:  instance
      - target_label:  __address__
        replacement:   <monitor-ip>:9115

Restart:

pgrep prometheus | xargs kill
./prometheus &

Install node_exporter on every server you want to monitor

sudo useradd --no-create-home --shell /bin/false nodeexp
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.1/node_exporter-1.8.1.linux-amd64.tar.gz
tar xf node_exporter-*.tar.gz
sudo mv node_exporter-*/node_exporter /usr/local/bin/
sudo tee /etc/systemd/system/node_exporter.service <<EOF
[Unit] Description=Node Exporter
[Service] User=nodeexp ExecStart=/usr/local/bin/node_exporter
[Install] WantedBy=multi-user.target
EOF
sudo systemctl enable --now node_exporter

Install Grafana (on the monitor VM)

sudo apt-get install -y adduser libfontconfig1 musl
wget https://dl.grafana.com/enterprise/release/grafana-enterprise_10.4.2_amd64.deb
sudo dpkg -i grafana-enterprise_10.4.2_amd64.deb
sudo systemctl enable --now grafana-server

Open http://<monitor-ip>:3000 → default login admin / admin → change password.

Wire Grafana to Prometheus

1. Connections → Data sources → Add data source → Prometheus.
2. URL = http://localhost:9090.
3. Save & Test → should say "Data source is working".

Import 3 ready-made dashboards

Dashboards → New → Import. Enter dashboard IDs:

ID	Dashboard
1860	Node Exporter Full (host metrics)
9964	Jenkins performance
7362	K8s cluster overview

Pick your Prometheus datasource → Import.

Add an alert (e.g., disk > 85%)

Alerting → Alert rules → New alert rule. Expression:

100 - ((node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100) > 85

Add a Contact point (SMTP or Slack) and a Notification policy.

What good looks like

You now have:

• One Grafana tab with CPU / RAM / disk on every VM.
• Another tab with Jenkins jobs, queue length, build times.
• Another with K8s pod counts, restarts, CPU per pod.

If any turns red, an email/Slack lands in your inbox.

Real-world win

The Trivy scan starts flagging vulnerable base images on Fridays at 3 AM (base image was updated by upstream). Your Grafana dashboard shows the build time crept from 8 → 22 minutes. Root cause: image pulls slowed down after a Docker Hub rate-limit change. You add a pull-through cache in Nexus → build times drop back to 8 min. This is real DevOps: measure, then fix.