Install Kubernetes Cluster - Adding Monitoring and Service Mesh for Observability (Part 2)

2
install kubernetes cluster - adding monitoring and service mesh
fresher-to-uber
Fresher To Uber

With the core cluster up and running in , it’s time to bring in some enterprise-grade tools using Helm. In this part, we’ll add monitoring with Prometheus and Grafana, service mesh capabilities with Linkerd.

Clone the repository

First you need to ssh to your control plane node and clone the repo that includes all the files you need for this setup.

git clone https://github.com/fresher-to-uber/k8s-cluster-setup.git
cd k8s-cluster-setup

Install Monitoring with Prometheus and Grafana

Monitoring is a must if you want to see what’s really happening inside your cluster. With , setting up Prometheus and Grafana is as easy as a couple of commands.

First, we need to install Helm

# Ensure correct architecture
export CLI_ARCH=$(dpkg --print-architecture)

# Add Helm
wget https://get.helm.sh/helm-v3.15.4-linux-$CLI_ARCH.tar.gz
tar -xf helm-v3.15.4-linux-$CLI_ARCH.tar.gz
sudo cp linux-$CLI_ARCH/helm /usr/local/bin/

Then install Prometheus stack with helm charts couldn't be easier

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

kubectl create ns monitoring
helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring

or simple just run the pre-defined script

$ bash monitoring/prometheus.sh

You can search the helm chart

Access Grafana

Modify grafana service type

kubectl edit svc prometheus-grafana -n monitoring -o yaml

Verify to make sure the change have applied

kubectl get svc prometheus-grafana -n monitoring

You should see the service type as NodePort and a port mapping like 80:3XXXX/TCP.

Deploy a test resource

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-prometheus-test
  labels:
    app: nginx-prometheus-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-prometheus-test
  template:
    metadata:
      labels:
        app: nginx-prometheus-test
    spec:
      containers:
      - name: nginx
        image: nginx:1.21
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-prometheus-test-service
  labels:
    app: nginx-prometheus-test
spec:
  selector:
    app: nginx-prometheus-test
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-prometheus-test-monitor
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: nginx-prometheus-test
  endpoints:
  - port: http 
    path: /metrics
    interval: 15s

If you have cloned the repo, you can find the file in monitoring/ folder. Then just run the command to apply the resource

kubectl apply -f monitoring/nginx-prometheusp-test.yaml

Access Grafana at http://<node-ip>:<node-port> using the default credentials (admin/prom-operator). Select one of the dashboards provided by prometheus

Install Service Mesh with Linkerd

Run each command below:

curl -sL run.linkerd.io/install | sh
export PATH=$PATH:$HOME/.linkerd2/bin
linkerd check --pre
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
linkerd check
linkerd viz install --set prometheus.enabled=false --set prometheusUrl=http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090 | kubectl apply -f -
linkerd viz check

Observe and make sure the output is success with green check mark.

Because we have our own Prometheus, so we need to configure our Prometheus instance to get Linkerd metrics. You can find the scrape config file in linkerd/ folder.

helm upgrade prometheus prometheus-community/kube-prometheus-stack -n monitoring -f linkerd/prometheus-scrape-configs.yaml

Edit Linkerd Dashboard to allow outside access

kubectl edit deploy web -n linkerd-viz

In spec.template.spec.containers.args section, set -enforced-host to empty

Next, change service type to NodePort, same thing we did to Grafana service

kubectl edit svc web -n linkerd-viz -o yaml

Now we can access Linkerd dashboard at http://<node-ip>:<node-port>.

Inject linkerd to the existing nginx-prometheus-test resource that we created in previous section.

kubectl get deploy nginx-prometheus-test -o yaml | \
linkerd inject - | kubectl apply -f -

Now we can observe that the pod is meshed in linkerd dashboard

Conclusion

Look at you, adding all these fancy enhancements to your Kubernetes cluster! Now, it’s monitoring, meshing, and ready to handle whatever comes its way (within reason). Sure, it’s a bit more complex, but it’s also way cooler. So go ahead—give yourself a pat on the back for leveling up your Kubernetes skills!

And again you can find the script .

2