What is self hosting and why should we do it?
Self hosting is the process of hosting your own services, instead of using a third party. For example, instead of using Google Drive, you can use Nextcloud to host your files. Or instead of using Netflix, you can use Plex to host your media.
It is a great way to reduce your reliance on third parties, and to take control of your data. It also helps to decentralize the internet, and makes the internet more resilient.
Prerequisites
- Basic knowledge of Docker and Docker Compose.
- Docker environment with Portainer installed (see this guide).
Abstract overview
We'll be setting up a Grafana dashboard (open source analytics & monitoring solution) as an example, using the grafana/grafana Docker image. We'll be using Cloudflare Tunnels to expose the Grafana dashboard to the internet, without exposing any ports.
Cloudflare Tunnel is tunneling software that lets you quickly secure and encrypt application traffic to any type of infrastructure, so you can hide your web server IP addresses, block direct attacks, and get back to delivering great applications.
Lastly, for data sources we'll be using Prometheus and cAdvisor images respectively.
Configuring Portainer Network
We need to create a separate network for Grafana, Prometheus and cAdvisor. To do this, we'll be using Portainer (container management tool).
Head over to Portainer, and click on the "Networks" tab. Then click on the "Add network" button.
Name the network web, and leave the rest of the settings as default. Click on the "Create the network" button.
Once the network is created, we can start configuring Grafana.
Configuring Grafana
In the sidebar, click on the "Stacks" tab. Then click on the "Add stack" button.
Name it monitoring, and set the stack file to the following:
version: '3.8'
networks:
# This is the network we created earlier
web:
external: true
volumes:
# These are the volumes we'll be using for storing data
prometheus_data: {}
grafana_data: {}
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
# These are the ports we'll be using for Prometheus
ports:
- 9090:9090/tcp
networks:
- web
volumes:
- /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
restart: unless-stopped
command:
# This is the configuration file for Prometheus
# We'll be editing this file later for setting up cAdvisor
- '--config.file=/etc/prometheus/prometheus.yml'
cadvisor:
image: zcube/cadvisor:latest
container_name: cadvisor
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
# These are the ports we'll be using for cAdvisor
ports:
- 8080:8080/tcp
networks:
- web
privileged: true
restart: unless-stopped
grafana:
image: grafana/grafana
container_name: grafana
volumes:
- grafana_data:/var/lib/grafana
# These are the ports we'll be using for Grafana
ports:
- 3000:3000/tcp
networks:
- web
restart: unless-stopped
Once this is done, click on the "Deploy the stack" button.
All the three services should be up and running now. You can check this by clicking on the "Containers" tab.
Configuring Prometheus
Now we need to configure Prometheus to scrape data from cAdvisor. To do this, we'll be editing the prometheus.yml file.
To do this, we'll need to access our server or machine using SSH. Once we've logged in, run the following command:
# We need to navigate to the prometheus directory
cd /etc/prometheus/
# Then we need to edit the prometheus.yml file
nano prometheus.yml
Now copy and paste the following configuration into the file:
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
# Job for cdvisor
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8080']
This will tell Prometheus to scrape data from cAdvisor. Once you're done, press Ctrl + X to exit the editor. Then press Y to save the changes.
Once all the changes are saved, go back to Portainer. Click on the "Stacks" tab, and then click on the "monitoring" stack.
Select the editor tab on the right, click on the "Update the stack" button (enable Re-pull image and redeploy) and finally press the "Update" button .
This should restart all the monitoring services, which should update the Prometheus configuration.
Configuring Cloudflare Tunnel
Now, we'll need to create a Cloudflare zero trust account (to authenticate with tunnels). To do this, go to Cloudflare Zero Trust.
Once you've created your account, you'll be able to create a tunnel. To do this, click on the "Create a tunnel" button.
Once you've created your tunnel, you'll be navigatd to the connector installation page. Here, you'll be able to download the connector for your operating system. For this guide, we'll be using the Docker connector.
Once the connector is installed, we can specify the grafana service (we have set the port to 3000) to expose.
After the tunnel is created, you'll be able to see the container inside portainer in the "Stacks" tab.
Accessing Grafana
To access Grafana, you'll need to navigate to https://<tunnel_name>.yourdomain.com/. For example, my tunnel name is grafana, we'll need to navigate to https://grafana.yashgarg.dev/.
You should be able to see the Grafana login page. The default username and password are both admin.
Setting Prometheus as Data Source
Once you've logged in, we need to setup Prometheus as a data source. To do this, click on the "Configuration" tab on the left, and then click on the "Data Sources" tab.
Set the name to Prometheus, and set the URL to http://prometheus:9090. Then click on the "Save & Test" button.
Now that we've set up Prometheus as a data source, we can add the cadvisor dashboard.
Adding cAdvisor Dashboard
To add the cAdvisor dashboard, click on the "Create" button on the left. Then click on the "Import" button.
Enter the dashboard ID as 14282, and click on the "Load" button. You should now see the cAdvisor dashboard.
It will start to populate the data after a few minutes, and should look like this after a few hours.
Conclusion
We have successfully setup a monitoring stack using Prometheus, cAdvisor, and Grafana. We've exposed the Grafana dashboard using Cloudflare Tunnels without exposing any ports externally.
Similarly, you can also use Cloudflare Tunnels to expose other services like qBittorrent, Plex, etc.
For reference, some of my commonly used docker-compose files can be found here.