8 min read

Visualizing Traefik Metrics and HTTP Logs in Grafana

Containerized Setup of Grafana, InfluxDB, Promtail and Grafana Loki to Visualize Traefik Metrics and HTTP Logs.
Visualizing Traefik Metrics and HTTP Logs in Grafana
Photo by Jonas Leupe / Unsplash

In my previous blog posts I outlined the necessary steps in order to create a monitoring dashboard with Grafana and various other software components such as Telegraf and InfluxDB. Moreover, I provided you guidance on how to use Grafana Loki as well as Promtail to visualize HTTP logs from Nginx Proxy Manager.

My older blog posts can be found here:

Monitoring Dashboard with Grafana, Telegraf, InfluxDB and Docker
Creating a dockerized Grafana monitoring dashboard to visualize statistics of your server and Docker containers.
Visualizing Logs with Grafana, Loki, Promtail and Docker
Creating a dockerized Grafana dashboard to visualize log data of the popular reverse proxy Nginx Proxy Manager (NPM) or any other logs.

As I've changed my IT infrastructure to run Traefik as reverse proxy, the need to change my Grafana monitoring dashboard slightly arised. Especially, as Traefik natively supports pushing metrics directly into a database like InfluxDB.


In today's blog post, I will provide you with detailed steps on how to activate metrics in Traefik and push them into a containerized InfluxDB database.

Moreover, we will enable logging of HTTP requests for Traefik and use Promtail to push them into Grafana Loki. Finally, we will configure the InfluxDB and Loki data sources in Grafana and import a pre-built monitoring dashboard by me to inspect Traefik metrics and HTTP logs.

The dashboard will look something like this in the end:

Traefik Metrics and HTTP Logs in Grafana

Configuring Container Services

This section contains the mandatory configuration steps for Traefik, InfluxDB as well as Promtail and Grafana Loki.

The general docker-compose.yml file we will be using is the following:

version: "3"


    image: influxdb:1.8.10
    container_name: influxdb
    hostname: influxdb
    restart: unless-stopped
      - /mnt/docker-volumes/influxdb/data:/var/lib/influxdb
      - /mnt/docker-volumes/influxdb/influxdb.conf:/etc/influxdb/influxdb.conf:ro
      - /mnt/docker-volumes/influxdb/init:/docker-entrypoint-initdb.d
      - INFLUXDB_ADMIN_PASSWORD=MyVeryStrongAdminPassword

      image: grafana/loki:latest
      hostname: loki
      container_name: loki
      - /mnt/docker-volumes/loki:/etc/loki
       - ""
      restart: unless-stopped
      user: 1000:1000
      command: -config.file=/etc/loki/loki-config.yml

    image: grafana/promtail:latest
    container_name: promtail
      - loki
    hostname: promtail
      - /var/log:/var/log
      - /mnt/docker-volumes/promtail:/etc/promtail
      - /mnt/docker-volumes/traefik/logs:/var/log/traefik
    restart: unless-stopped
    command: -config.file=/etc/promtail/promtail-config.yml

    image: grafana/grafana:latest
    container_name: grafana
    hostname: grafana
    restart: unless-stopped
    user: 1000:1000 # please adjust
      - influxdb
      - loki
      - /mnt/docker-volumes/grafana:/var/lib/grafana
      - GF_SERVER_ROOT_URL=http://grafana.example.com # please adjust
      - 3000:3000


As can be seen in the above YML file, we have configured various volume mounts holding our configuration files for the services. In my case, all volume mounts are located at /mnt/docker-volumes/<container-name>. Adjust to your infrastructure setup if needed.

In the following we will discuss the necessary configuration steps for all container services.


For this blog post I will not go into detail on how to configure and setup a full-blown, dockerized Traefik instance. If you do not have a Traefik instance yet, may head over to my GitHub repository, which holds all necessary things.

However, in order to push Traefik metrics into an InfluxDB database, you must adjust your static configuration file of Traefik. Add the following code snippet:

    address: http://influxdb:8086
    protocol: http
    database: traefik
    username: traefikuser
    password: MyVeryStrongInfluxdbPassword
    addRoutersLabels: true
    addServicesLabels: true
    pushInterval: 60s

enabling Traefik metrics - traefik.yml

Furthermore, we have to enable access logs for Traefik. To enable access logs for Traefik, adjust your static configuration file again and add the following lines:

  filePath: "/logs/traefik.log"
  format: json
      #- "200"
      - "400-599"
  # collect logs as in-memory buffer before writing into log file
  bufferingSize: 0
      defaultMode: drop # drop all headers per default
          User-Agent: keep # log user agent strings  

enabling Traefik access logs - traefik.yml

Note that we actively choose JSON formatting over CLM, as JSON logs contain more details than CLM formatted logs. Don't ask me why but it is what it is. Furthermore, we log errors only. May adjust if needed.Do not forget to configure log rotations for Traefik. An example script can be found in my GitHub repository here.

That's basically it for configuring the Traefik reverse proxy.


As we have enabled metrics for Traefik, which will be later pushed into an InfluxDB database, we must setup such database of course. The above docker-compose.yml already holds our necessary InfluxDB container service.

However, besides that, we also have to configure two additional configuration files:

  • influxdb.conf
  • /init/create-traefik.iql

The influxdb.conf file will be stored in the root directory of the volume mount for InfluxDB. The configuration file holds the following entries (nothing to adjust):

# Bind address to use for the RPC service for backup and restore.
bind-address = ""
  dir = "/var/lib/influxdb/meta"
  dir = "/var/lib/influxdb/data"
  wal-dir = "/var/lib/influxdb/wal"
  series-id-set-cache-size = 100
  enabled = true
  bind-address = ":8086"
  auth-enabled = true


Besides that we have to create an initialization file, which will be run when the container starts. We will utilize this startup run feature to initialize our database. The init configuration file will be stored within the init volume mount directory.

It can be named randomly but must have the extension .iql. I've named my configuration file create-traefik.iql and it holds the following data:

CREATE USER traefikuser WITH PASSWORD 'MyVeryStrongInfluxdbPassword'
GRANT ALL ON traefik to traefikuser


Note that this password is referenced in your static configuration file of Traefik above. So if you change it here, also change it in your final Traefik config!


In order for our Promtail container to access, read and parse log data, we must specify where our log data is available and which of them should be parsed. The log data must be mounted inside the Promtail container, which we defined in the docker-compose.yml file above.

Specifying which log file to parse is done through a promtail-config.yml configuration file, located at /mnt/docker-volumes/promtail/promtail-config.yml in my case.

For our use case, I'll force Promtail to parse the following two logs:

  1. Auth logs: We've successfully bind mounted the logs of my Linux server at /var/log into the Promtail container. Therefore, let's use it! I want to parse the well-known auth.log log file that holds many interersting things such as SSH logins etc.
  2. Traefik logs: Additionally, I want to parse the log files of Traefik. Access logging was enabled previously in the static configuration file of Traefik. The log file will be stored in the directory /logs of your Traefik container volume.

The configuration file should look like this (nothing to adjust):

  http_listen_port: 9080
  grpc_listen_port: 0

  filename: /tmp/positions.yaml

  - url: http://loki:3100/loki/api/v1/push

# local machine logs
  - job_name: auth
    - targets:
        - localhost
        job: authlogs
        __path__: /var/log/auth.log

  - job_name: traefik
    - targets:
        - localhost
        job: traefiklogs
        __path__: /var/log/traefik/*.log


Note that the path /var/log/traefik is dependent on the used docker volume mount. In the above defined docker-compose.yml, we mounted our Traefik logs as well as auth logs into the Promtail container.


Finally, we must also define a configuration file for Loki. The file is called loki-config.yml and defined in the above docker-compose.yml. In my case, it is located at /mnt/docker-volumes/loki/loki-config.yml.

The config file should contain the following (nothing to adjust):

auth_enabled: false

  http_listen_port: 3100
  grpc_listen_port: 9096

  path_prefix: /tmp/loki
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
      store: inmemory

   reject_old_samples: true
   reject_old_samples_max_age: 168h
   retention_period: 360h
   max_query_series: 100000
   max_query_parallelism: 2
   split_queries_by_interval: 0      

    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
        prefix: index_
        period: 24h

  parallelise_shardable_queries: false

  max_concurrent: 2048

  max_outstanding_per_tenant: 4096
  compress_responses: true

  alertmanager_url: http://localhost:9093


Spawning Our Docker Containers

If you successfully adjusted the above docker-compose.yml file to your needs and ensured that all configuration files for InfluxDB, Promtail and Loki exist, we will now be able to proceed booting up our Docker stack of multiple containers.

A single Linux command from the directory your docker-compose.yml is located and your containers should start to see daylight:

sudo docker compose up -d

If everything went well, you should now be able to log into your Grafana instance at http://<your-servers-ip-address>:3000 via a web browser.

The default username and password is admin.
If you are not greeted by Grafana, please inspect your Docker logs to identify the misconfiguration (e.g. permissions). Sometimes it also takes a while until all containers are up and running. Depending on your server's hardware, give it a few minutes to come up.

Configuring Grafana

As soon as the Grafana instance is ready, we must complete our last step of adding data sources as well as creating a new dashboard.

Defining Data Sources

Upon logging into your fresh Grafana instance via a web browser, we must specify our data sources.

Since all docker containers are started from a single docker-compose.yml file, the containers will be put inside the same Docker network. Therefore, we can easily use the container names instead of IP addresses.

Head over to the URL /datasources and choose Loki as data source. Configure it as follows:

Defining Loki as new data source, no auth required

Additionally, configure another datasource for InfluxDB. Head over to the URL /datasources again and now choose InfluxDB as data source. You must define an InfluxDB URL as well as the database, user and password for authentication.

We previously defined an init-script for InfluxDB. In this script, we defined our user traefikuser as well as the password MyVeryStrongInfluxdbPassword. Use these credentials, together with the defined database name traefik, when adding InfluxDB as data source.If you changed those credentials, use the new ones! 
Adding InfluxDB as new data source - specifying the URL

Importing a Dashboard

Finally, we have everything in place to start graphing. Grafana is up and running and we successfully configured InfluxDB and Loki as data sources to pull our data points from. Since creating a Grafana dashboard is kinda time consuming, I have uploaded my template here. Feel free to use it as a starting point:

Browse to the Grafana URL /dashboard/import and upload the above JSON file. Select the configured InfluxDB and Loki sources as default data sources. You should then be redirected to your newly added dashboard automatically, which hopefully displays all statistics correctly.

In case your newly added Grafana dashboard does not instantly display graphs and data, give the Docker containers a few minutes. If you are unsure whether the whole setup is running correctly, inspect the logs of all containers for troubleshooting.