Monitoring Dashboard with Grafana, Telegraf, InfluxDB and Docker
Creating a dockerized Grafana monitoring dashboard to visualize statistics of your server and Docker containers.
Dashboards are great. They look cool, fancy and especially allow a fast visual analysis of various statistics like CPU utilization, disk usage, network stuff and many more things. Numerous free and open-source software (FOSS) exist that can collect your desired data points, send them to a prefered storage solution - from where they can be finally pulled for data filtering, visualization and many more things.
In today's blog post, I will guide you through the steps of setting up a dockerized tech stack consisting of Grafana, Telegraf and InfluxDB. We will use Grafana for handling the visual stuff like the actual dashboard with graphs, timelines etc. Telegraf will be used for data collection, basically retrieving all the interesting data points like CPU utilization, Docker stats etc. And InfluxDB will finally be utilized to store our collected data and to allow Grafana to use it as data source.
For example all necessary config files as well as the final docker-compoye.yml to get you started fast. Just a quick docker compose up and your Grafana dashboard is not far away.
Come back later to this blog post if you feel lost.
The end goal may look something like this:
Creating Our Docker-Compose.yml
A Docker compose file defines all necessary information to spawn our Docker containers with persistent data storage. Note that we'll use bind volumes to persist our container data. I personally store all my Docker volume data at the path /mnt/docker-volumes/<docker-container-name> of my Linux server.
Please adjust the configuration to your needs and infrastructure setup. Especially watch out for:
uid and gid mappings; especially for Telegraf!
Grafana's server URL as env variable with your server's IP or your subdomain. Might be optional.
The Telegraf 1.20.3 release changed the official Telegraf DockerHub image to no longer run the Telegraf service as root. With this change, the Telegraf service runs with the least amount of privileges in the container to enhance security given the wide extensibility and array of plugins available in Telegraf.
If a user passes in the Docker socket for Telegraf to monitor Docker itself, then they will need to add the telegraf user to the group that owns the Docker socket. One way to achieve this is to obtain the GUID first and define it in the user parameter of Telegraf's docker-compose.yml. It should then look something like this: user: telegraf:998
The numeric GUID can be obtained by the following command run on your server: stat -c '%g' /var/run/docker.sock
Note that we actively bind mount various paths of our Docker server into the Telegraf container. This ensures that we are retrieving stats like CPU, disks, network etc. from our server instead of the Telegraf docker container.
As visible in the above docker-compose.yml, we bind mounted a so called influxdb.conf and an init directory into the InfluxDB docker container. The configuration file influxdb.conf must be present on our server before actually starting the InfluxDB docker container. Furthermore, we must define another file inside the init directory to automatically create a new InfluxDB database and user.
Use the following influxdb.conf and make sure to correctly bind mount it into the Docker container:
Use the following create-telegraf.iql configuration file and place it on your server host at the defined bind mount /mnt/docker-volumes/Influxdb/init/.
Note that if you change any credentials here, you have to reflect those in the below Telegraf configuration at [[outputs.influxdb]] as well!
As visible in the above docker-compose.yml, we bind mounted a so called telegraf.conf into the Telegraf docker container. This configuration file must be present on our server before actually starting the Telegraf docker container.
Use the following telegraf.conf and make sure to correctly bind mount it into the Docker container's location at /etc/telegraf/telegraf.conf. Please adjust to your needs. Telegraf is your data collector, so anything defined here will be later available in Grafana for graphing!
Spawning Our Docker Containers
If you successfully adjusted the above docker-compose.yml file to your needs and ensured that a correct Telegraf and InfluxDB configuration exists in advance, we will now be able to proceed booting up our Docker containers.
A single Linux command from the directory your docker-compose.yml is located and your containers should start to see daylight:
sudo docker-compose up -d
If everything went well, you should now be able to log into your Grafana instance at http://<your-servers-ip-address>:3000 via a web browser. If you are not greeted by Grafana, please inspect your Docker logs to identify the misconfiguration. Sometimes it takes a while until all containers are up and running. Depending on your server's hardware, give it a few minutes to come up.
The default username and password is admin.
Creating our Grafana Dashboard
After successfully logging into our Grafana instance and changing the default password, the first thing to do is adding InfluxDB as data source.
Adding a new data source
Jump into Grafana's settings located at /datasources and select InfluxDB as data source. You must define InfluxDB's URL as well as the database, user and password for authentication. Then proceed and hit Save & Test.
Since all docker containers are started from a single docker-compose.yml file, the containers will be put inside the same Docker network. Therefore, we can easily use the container's name instead of IP addresses. Therefore, just specify http://influxdb:8086 as URL.
We previously defined an init-script for InfluxDB. In this script, we defined our user telegrafuser as well as the password MyVeryStrongTelegrafUserPassword. Use these credentials, together with the defined database name telegraf, when adding InfluxDB as data source. If you changed those credentials, use the new ones!
Creating a new dashboard
Finally, we have everything in place to start graphing. Grafana is up and running and we successfully configured InfluxDB as data source to pull our data points from. Since creating a Grafana dashboard is kinda time consuming, I have uploaded my template here. Feel free to use it as a starting point:
Browse to the Grafana URL /dashboard/import and upload the above Grafana_Dashboard_Template.json file. Select InfluxDB as default data source. You should then be redirected to your newly added dashboard automatically, which hopefully displays all statistics correctly.
In case your newly added Grafana dashboard does not instantly display graphs and data, give the Docker containers a few minutes. Telegraf runs every 30s and pumps its collected data into InfluxDB.
If you are unsure whether the whole setup is running correctly, inspect the logs of Telegraf and InfluxDB for troubleshooting.
Enjoy and also have a look at:
You might also like...
Storing Secrets Securely and Encrypted using Git-Crypt
7 min read
Configuring SSO for Immich with Authelia OIDC
3 min read
Protecting Web Services with Authentik, Traefik and Azure AD
15 min read
Identifying Blind XSS Attack Vectors with XSSHunter
5 min read
Visualizing Traefik Metrics and HTTP Logs in Grafana