In our previous blog posts we outlined how to implement CrowdSec in conjunction with Traefik as reverse proxy. Furthermore, we showcased how to parse Traefik logs via Promtail and piping those logs into Grafana + Loki.

Configuring CrowdSec with Traefik

Utilizing CrowdSec and its Cyber Threat Intelligence (CTI) to ban malicious threat actors probing our exposed HTTP services in a collaborative manner.

LRVT's Security BlogLRVT

Visualizing Logs with Grafana, Loki, Promtail and Docker

Creating a dockerized Grafana dashboard to visualize log data of the popular reverse proxy Nginx Proxy Manager (NPM) or any other logs.

LRVT's Security BlogLRVT

Visualizing Traefik Metrics and HTTP Logs in Grafana

Containerized Setup of Grafana, InfluxDB, Promtail and Grafana Loki to Visualize Traefik Metrics and HTTP Logs.

LRVT's Security BlogLRVT

In todays blog post, we will focus on parsing CrowdSec decisions and building a selfhosted Cyber Threat Intelligence (TI) dashboard using Grafana + VictoriaMetrics. The dashboard will offer threat intelligence data, automatic hyperlinks to CrowdSec's CTI, Shodan, Censys and more external services as well as a world map of attacking IPs and countries.

Prerequisites

For this blog post I will assume that you are already running CrowdSec as well as a Grafana instance.

It does not really matter whether you have combined CrowdSec with Traefik or any other reverse proxy. Moreover, CrowdSec can be run as Docker container or be installed as baremetal service.

During the blog post, we will spawn a new container service called VictoriaMetrics (VM) and adjust some CrowdSec configuration files. Finally, we import an already existing Grafana dashboard, which visualizes CrowdSec's Threat Intelligence data.

Spawning VictoriaMetrics

CrowdSec itself already provides a Prometheus HTTP endpoint to obtain logs and stats about decisions. However, the data provided is quite limited and cannot be compared to the advanced details available on the proprietary CrowdSec Console dashboard at https://app.crowdsec.net/.

Due to this reason, we will spawn VictoriaMetrics (VM). VM provides us with a selfhosted monitoring solution similar to Prometheus, offering high performance, efficient storage, and compatibility with existing Prometheus ecosystems.

We will use it as data broker. Taking JSON data, feeded from our CrowdSec instance and making the data available as Prometheus API to Grafana.

Docker Compose for VM

VictoriaMetrics can be easily spawned using Docker Compose.

Please ensure to put the container service in the same Docker network as your running Grafana and CrowdSec instances. This way, you can utilize Docker networking and must not map or expose any container ports of VictoriaMetrics to your underlying host. In my case, the external Docker network is called crowdsec_grafana.

services:

  victoriametrics:
    image: victoriametrics/victoria-metrics:v1.115.0
    expose:
      - 8428 # VM UI
    volumes:
      - vmdata:/storage
    command:
      - "--storageDataPath=/storage"
      - "--httpListenAddr=:8428"
    restart: always
    networks:
      - crowdsec_grafana  

networks:
  crowdsec_grafana:
    external: true

volumes:
  vmdata: {}    

Afterwards, you can spawn VM via the following means:

docker compose up -d

And that's it. VictoriaMetrics is running in its own Docker network and waits for data points feeded in via the http://victoriametrics:8428/api/v1/import URL. No authentication requiered and only exposed to other containers running in the same network crowdsec_grafana.

Adjusting CrowdSec

In order to feed decision data into VictoriaMetrics, we must adjust the active configuration of our CrowdSec instance. We will create a new notification handler called victoriametrics.yaml and adjust our profiles.yaml to use the new handler.

If you run CrowdSec as Docker container (e.g. by following my previous blog post), you can find the configuration files at your volume bind mounts that were defined in the CrowdSec Docker Compose project.

In general, the data is typically stored at the following areas:

• Within CrowdSec's Docker container:
  • /etc/notifications/victoriametrics.yaml
  • /etc/profiles.yaml
• At CrowdSec's volume bind mounts (based on my blog posts):
  • /mnt/docker-volumes/crowdsec/etc/notifications/victoriametrics.yaml
  • /mnt/docker-volumes/crowdsec/etc/profiles.yaml
• For baremetal installations:
  • /etc/crowdsec/notifications/victoriametrics.yaml
  • /etc/crowdsec/profiles.yaml

Creating a New Notification Handler

The first step is creating a new notification handler.

For this, we will create a new file called victoriametrics.yaml within CrowdSec's notification directory. The file must have the following content:

type: http
name: http_victoriametrics
log_level: info
format: >
  {{- range $Alert := . -}}
  {{- range .Decisions -}}
  {"metric":{"__name__":"cs_lapi_decision","instance":"my-instance","country":"{{$Alert.Source.Cn}}","asname":"{{$Alert.Source.AsName}}","asnumber":"{{$Alert.Source.AsNumber}}","latitude":"{{$Alert.Source.Latitude}}","longitude":"{{$Alert.Source.Longitude}}","iprange":"{{$Alert.Source.Range}}","scenario":"{{.Scenario}}","type":"{{.Type}}","duration":"{{.Duration}}","scope":"{{.Scope}}","ip":"{{.Value}}"},"values": [1],"timestamps":[{{now|unixEpoch}}000]}
  {{- end }}
  {{- end -}}
url: http://victoriametrics:8428/api/v1/import
method: POST
headers:
  Content-Type: application/json
  # if you use vmauth as proxy, please uncomment next line and add your token
  #Authorization: "<SECRET-TOKEN>"

That's it. Via this notification handler, CrowdSec will happily format the threat intel data from decisions as JSON and pass it along to our VictoriaMetrics container via its API endpoint running on TCP/8428.

Adjusting our CrowdSec Profile

Finally, we have to adjust our profiles.yaml in CrowdSec to enable our new notification handler. For this, we will open the profiles YAML file and add our new notification handler named http_victoriametrics at the key notifications.

The file should look like this:

name: default_ip_remediation
#debug: true
filters:
 - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
 - type: ban
   duration: 4h
#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)
notifications:
 - http_victoriametrics
on_success: break
---
name: default_range_remediation
#debug: true
filters:
 - Alert.Remediation == true && Alert.GetScope() == "Range"
decisions:
 - type: ban
   duration: 4h
#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)
notifications:
 - http_victoriametrics
on_success: break

Afterwards, we must restart CrowdSec:

# restarting via docker cli
docker container restart crowdsec

# restarting via docker compose
docker compose -f <path-to-crowdsec-compose> up -d --force-recreate

# restarting via systemctl on baremetal
systemctl restart crowdsec

Installing GeoIP-Enrich Parser

Most CrowdSec instances come with the geoip-enrich parser installed as default. This one is necessary as we want to enrich banned IP addresses with geolocation data to display a worldmap in our Grafana dashboard later.

Although installed per default, we will verify that this is really the case for your instance. To do so, issue the following command:

# install parser via docker cli
docker exec -it crowdsec cscli parsers install crowdsecurity/geoip-enrich

# install parser via cscli on baremetal
cscli parsers install crowdsecurity/geoip-enrich

Typically, it will report back Nothing to do - as already installed.

Setup Grafana Dashboard

If you already run your own Grafana instance, adding the new CrowdSec Threat Intelligence dashboard is very simple. You just have to import the already existing dashboard from GrafanaLabs.

Crowdsec Cyber Threat Insights | Grafana Labs

Grafana Labs

Before doing so, we will add VictoriaMetrics as new data source though.

If you do not run Grafana yet, you may use this compose example:

services:

  grafana:
    image: grafana/grafana-oss:latest    
    container_name: grafana
    restart: unless-stopped    
    user: 1000:1000 # your UID/GID
    environment:
      - GF_SERVER_ROOT_URL=https://grafana.example.com # your FQDN
    volumes:
      - ./volume-data/grafana:/var/lib/grafana
    ports:
      - 3000:3000
    expose:
      - 3000
    networks:
      - crowdsec_grafana  

networks:
  crowdsec_grafana:
    external: true

Add VictoriaMetrics as Data Source

1. Log into your Grafana instance as admin
2. Browse the datasources area at /connections/datasources
3. Hit the button Add new datasource
4. Chose Prometheus
5. Define a name for your datasource; e.g. victoriametrics
6. Define the URL http://victoriametrics:8428
7. Hit Save and test

Importing the Dashboard

After adding VictoriaMetrics as new datasource, we can proceed with importing the CrowdSec Threat Intelligence dashboard.

To do so, follow these steps:

1. Log into your Grafana instance as admin
2. Browse the dashboards area at /dashboards
3. Hit the buttons New > Import
4. Define the dashboard ID 21689 from GrafanaLabs and hit Load
5. Define a dashboard name like Crowdsec Cyber Threat Insight
6. Define the previously created victoriametrics data source
7. Hit the button Import

And that's basically it. You can now browse the newly imported dashboard and enjoy your CrowdSec Threat Intelligence feed. It may take some time for CrowdSec and the new notification handler to pass decision data into VictoriaMetrics. So take a break, get some coffee and check the dashboard at a later time again.

The Extra Mile

I've gone some extra steps and combined the dashboard with my already existing dashboards from other blog posts referenced above.

Now, I have a large Grafana dashboard that provides:

• Monitoring my Proxmox Ubuntu VM (resources, uptime, disks etc.)
• Monitoring my running Docker containers (CPU usage, memory usage etc.)
• Parsing and filtering Traefik and OS logs via Loki (http, ssh etc.)
• Inspecting CrowdSec Threat Intelligence (TI) data