Monitoring with Grafana and Victoria Metrics

Author: pluralplay, kutovoys
Deployment of an intelligent monitoring stack, including Grafana for creating dynamic dashboards, Victoria Metrics for storing and processing metrics, Node Exporter for monitoring servers, cAdvisor for analyzing containers, Xray Checker for controlling network connections, and Netbird for building a secure network architecture. This solution provides maximum transparency, performance, and security for your infrastructure.

Important

This guide describes the installation of services on a separate monitoring VPS server. The steps are described for a root user.

• Grafana - an instrument that transforms various forms of metrics received from Victoria Metrics into visual forms for analytical and system monitoring.

• Victoria Metrics (VMetrics) - is directly a system for collecting and storing metrics from all machines received from the agent, vmagent - a carrier of these metrics to vmetrics.

• Node Exporter - an instrument for collecting metrics. It collects information about various resources of the node, such as processor load, memory usage, disk input-output, network statistics, and others.

• cAdvisor - also an instrument for collecting metrics. It collects information about the performance of Docker containers.

• Xray-Checker - checks the state of your hosts and sends metrics to Victoria Metrics.

Scheme of work

Configuration of your server with Grafana and Victoria Metrics

Important

Issuing certificates for your domain is not considered in this article. Certificates for the domain must be placed in /opt/monitoring/nginx, or passed as a volume where the certificates are located, for example in /etc/letsencrypt/live/yourdomain.com.

Note

Don’t forget to add a DNS record pointing to this server for the domain, for example grafana.yourdomain.com which will be used in GF_SERVER_DOMAIN and GF_SERVER_ROOT_URL.

Note

Don’t forget to add this server to the NetBird network, as this is how it is done, you can look here.

1. Docker installation:

   curl -fsSL https://get.docker.com | sh

2. Create directories and docker-compose.yml:

   mkdir -p /opt/monitoring/{cadvisor,nodeexporter,nginx,vmagent/conf.d} && cd /opt/monitoring && nano docker-compose.yml

   Insert the following content and replace the fields with your own:

   docker-compose.yml

   services:
     vmsingle:
       image: victoriametrics/victoria-metrics:v1.124.0
       container_name: vmsingle
       hostname: vmsingle
       restart: unless-stopped
       volumes:
         - ./victoria-metrics-data:/victoria-metrics-data
       command:
         - '--storageDataPath=/victoria-metrics-data'
       networks:
         - grafana-network
       ports:
         - IP:8428:8428 # Here we specify the NetBird address (in this case the Grafana server)
   
     grafana:
       image: grafana/grafana:12.2.0-17027759091
       container_name: grafana
       hostname: grafana
       restart: unless-stopped
       volumes:
         - grafana-data:/var/lib/grafana
         - ./grafana/provisioning:/etc/grafana/provisioning
       environment:
         - GF_SERVER_DOMAIN=yourdomain.com # The main domain before Grafana, don't forget to add a DNS record for it (if you mounted a subdomain, you need to write the target domain here)
         - GF_SERVER_ROOT_URL=https://grafana.yourdomain.com/grafana # Full path to Grafana
         - GF_SERVER_SERVE_FROM_SUB_PATH=true
         - GF_SERVER_HTTP_PORT=3000
         - GF_SERVER_PROTOCOL=http
         - GF_SECURITY_ADMIN_USER=admin
         - GF_SECURITY_ADMIN_PASSWORD=admin
         - GF_ANALYTICS_REPORTING_ENABLED=false
       networks:
         - grafana-network
   
     xray-checker:
       image: kutovoys/xray-checker
       container_name: xray-checker
       hostname: xraychecker
       restart: unless-stopped
       environment:
         - "SUBSCRIPTION_URL=https://subscription_link" # Here we specify the subscription link for XrayChecker
         - "PROXY_STATUS_CHECK_URL=http://google.com/generate_204"
         - "PROXY_CHECK_INTERVAL=60"
       ports:
         - 127.0.0.1:2112:2112
       networks:
         - grafana-network
   
     grafana-nginx:
       image: nginx:1.28
       container_name: grafana-nginx
       volumes:
           - ./nginx/nginx.conf:/etc/nginx/conf.d/default.conf:ro
           - /etc/letsencrypt/live/yourdomain.com/fullchain.pem:/etc/nginx/ssl/yourdomain.com/fullchain.pem:ro
           - /etc/letsencrypt/live/yourdomain.com/privkey.pem:/etc/nginx/ssl/yourdomain.com/privkey.pem:ro
       restart: unless-stopped
       ports:
         - 443:443
       networks:
         - grafana-network
   
   volumes:
     grafana-data:
   
   networks:
     grafana-network:
       name: grafana-network
       driver: bridge
       external: false

3. Create nginx.conf:

   nano /opt/monitoring/nginx/nginx.conf

4. Insert the following content:

   nginx.conf

   ssl_protocols TLSv1.2 TLSv1.3;
   ssl_ecdh_curve X25519:prime256v1:secp384r1;
   ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;
   ssl_prefer_server_ciphers on;
   ssl_session_timeout 1d;
   ssl_session_cache shared:MozSSL:10m;
   ssl_session_tickets off;
   
   server {
       listen 443 ssl;
       http2 on;
       server_name grafana.yourdomain.com; # Write the domain, or subdomain for external access to grafana
   
       ssl_certificate "/etc/nginx/ssl/yourdomain.com/fullchain.pem"; # Path to the certificate
       ssl_certificate_key "/etc/nginx/ssl/yourdomain.com/privkey.pem"; # Path to the private key
       ssl_trusted_certificate "/etc/nginx/ssl/yourdomain.com/fullchain.pem"; # Path to the certificate
   
       location /grafana {
           proxy_pass http://grafana:3000;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
       }
   }

5. Run docker-compose

   docker compose up -d && docker compose logs -f -t

Installation and configuration of the Grafana server

1. Installation of cadvisor, node_exporter, vmagent:

   cd /opt/monitoring/cadvisor && wget https://github.com/google/cadvisor/releases/download/v0.53.0/cadvisor-v0.53.0-linux-amd64 && mv cadvisor-v0.53.0-linux-amd64 cadvisor && chmod +x cadvisor && cd /opt/monitoring/nodeexporter && wget https://github.com/prometheus/node_exporter/releases/download/v1.9.1/node_exporter-1.9.1.linux-amd64.tar.gz && tar -xvf node_exporter-1.9.1.linux-amd64.tar.gz && cd node_exporter-1.9.1.linux-amd64 && mv node_exporter /opt/monitoring/nodeexporter && cd .. && chmod +x node_exporter && rm -f node_exporter-1.9.1.linux-amd64.tar.gz && rm -r node_exporter-1.9.1.linux-amd64 && cd /opt/monitoring/vmagent && wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.123.0/vmutils-linux-amd64-v1.123.0.tar.gz && tar -xvf vmutils-linux-amd64-v1.123.0.tar.gz && mv vmagent-prod vmagent && find . ! -name 'vmagent' -type f -delete && chmod +x vmagent && cd

2. Create vmagent configuration file:

   nano /opt/monitoring/vmagent/scrape.yml

   Insert the following content:

   scrape.yml

   scrape_config_files:
     - "/opt/monitoring/vmagent/conf.d/*.yml"
   
   global:
     scrape_interval: 15s

3. Create cadvisor configuration file:

   nano /opt/monitoring/vmagent/conf.d/cadvisor.yml

   Insert the following content:

   cadvisor.yml

   - job_name: integrations/cAdvisor
     scrape_interval: 15s
     static_configs:
       - targets: ['localhost:9101']
         labels:
           instance: "your_instance_name(server Grafana)"

4. Create node_exporter configuration file:

   nano /opt/monitoring/vmagent/conf.d/nodeexporter.yml

   Insert the following content:

   nodeexporter.yml

   - job_name: integrations/node_exporter
     scrape_interval: 15s
     static_configs:
       - targets: ['localhost:9100']
         labels:
           instance: "your_instance_name(server Grafana)"

5. Create xray checker configuration file:

   nano /opt/monitoring/vmagent/conf.d/xraychecker.yml

   Insert the following content:

   xraychecker.yml

   - job_name: xray_checker
     scrape_interval: 15s
     static_configs:
       - targets: ['127.0.0.1:2112']
         labels:
           instance: "your_instance_name(server Grafana)"

6. Create cadvisor service file:

   nano /etc/systemd/system/cadvisor.service

   Insert the following content:

   cadvisor.service

   [Unit]
   Description=cAdvisor
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=root
   Group=root
   Type=simple
   ExecStart=/opt/monitoring/cadvisor/cadvisor \
           -listen_ip=127.0.0.1 \
           -logtostderr \
           -port=9101 \
           -docker_only=true
   Restart=always
   RestartSec=5
   
   [Install]
   WantedBy=multi-user.target

7. Create node_exporter service file:

   nano /etc/systemd/system/nodeexporter.service

   Insert the following content:

   nodeexporter.service

   [Unit]
   Description=Node Exporter
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=root
   Group=root
   Type=simple
   ExecStart=/opt/monitoring/nodeexporter/node_exporter --web.listen-address=127.0.0.1:9100
   Restart=always
   RestartSec=5
   
   [Install]
   WantedBy=multi-user.target

8. Create vmagent service file:

   nano /etc/systemd/system/vmagent.service

   Attention

   In the line -remoteWrite.url=http://IP:8428/api/v1/write you need to replace with the address of Netbird (in this case the Grafana server)

   Insert the following content:

   vmagent.service

   [Unit]
   Description=VictoriaMetrics Agent
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=root
   Group=root
   Type=simple
   ExecStart=/opt/monitoring/vmagent/vmagent \
         -httpListenAddr=127.0.0.1:8429 \
         -promscrape.config=/opt/monitoring/vmagent/scrape.yml \
         -promscrape.configCheckInterval=60s \
         -remoteWrite.url=http://IP:8428/api/v1/write
   Restart=always
   RestartSec=5
   
   [Install]
   WantedBy=multi-user.target

9. Start services and add them to autostart:

   systemctl daemon-reload && systemctl enable cadvisor nodeexporter vmagent && systemctl start cadvisor nodeexporter vmagent

10. Check service status:

    systemctl status cadvisor
    systemctl status nodeexporter
    systemctl status vmagent

Installation and configuration of the server with the Remnawave panel

1. Create directories and install cadvisor, node_exporter, vmagent:

   mkdir -p /opt/monitoring/{cadvisor,nodeexporter,vmagent/conf.d} && cd /opt/monitoring/cadvisor && wget https://github.com/google/cadvisor/releases/download/v0.53.0/cadvisor-v0.53.0-linux-amd64 && mv cadvisor-v0.53.0-linux-amd64 cadvisor && chmod +x cadvisor && cd /opt/monitoring/nodeexporter && wget https://github.com/prometheus/node_exporter/releases/download/v1.9.1/node_exporter-1.9.1.linux-amd64.tar.gz && tar -xvf node_exporter-1.9.1.linux-amd64.tar.gz && cd node_exporter-1.9.1.linux-amd64 && mv node_exporter /opt/monitoring/nodeexporter && cd .. && chmod +x node_exporter && rm -f node_exporter-1.9.1.linux-amd64.tar.gz && rm -r node_exporter-1.9.1.linux-amd64 && cd /opt/monitoring/vmagent && wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.123.0/vmutils-linux-amd64-v1.123.0.tar.gz && tar -xvf vmutils-linux-amd64-v1.123.0.tar.gz && mv vmagent-prod vmagent && find . ! -name 'vmagent' -type f -delete && chmod +x vmagent && cd

2. Create vmagent configuration file:

   nano /opt/monitoring/vmagent/scrape.yml

   Insert the following content:

   scrape.yml

   scrape_config_files:
     - "/opt/monitoring/vmagent/conf.d/*.yml"
   
   global:
     scrape_interval: 15s

3. Create cadvisor configuration file:

   nano /opt/monitoring/vmagent/conf.d/cadvisor.yml

   Insert the following content:

   cadvisor.yml

   - job_name: integrations/cAdvisor
     scrape_interval: 15s
     static_configs:
       - targets: ['localhost:9101']
         labels:
           instance: "your_instance_name(server Remnawave panel)"

4. Create node_exporter configuration file:

   nano /opt/monitoring/vmagent/conf.d/nodeexporter.yml

   Insert the following content:

   nodeexporter.yml

   - job_name: integrations/node_exporter
     scrape_interval: 15s
     static_configs:
       - targets: ['localhost:9100']
         labels:
           instance: "your_instance_name(server Remnawave panel)"

5. Create remnawave panel configuration file:

   nano /opt/monitoring/vmagent/conf.d/remnawave.yml

   Insert the following content:

   Important

   Replace username and password with your data, which are located at /opt/remnawave/.env (section ### PROMETHEUS ###)

   Example

   .env

   ### PROMETHEUS ###
   ### Metrics are available at /api/metrics
   METRICS_USER=gjlToiPH
   METRICS_PASS=VsCNutHF

   remnawave.yml

   - job_name: remnawave
     scrape_interval: 15s
     static_configs:
       - targets: ['127.0.0.1:3001']
         labels:
           instance: "your_instance_name(server Remnawave panel)"
     basic_auth:
       username: "xxxxxxx"
       password: "xxxxxxx"

6. Forward the metrics port in docker-compose.yml:

   nano /opt/remnawave/docker-compose.yml

   Find the following content and add:

   docker-compose.yml

   remnawave:
     image: remnawave/backend:latest
     container_name: remnawave
     hostname: remnawave
     restart: always
     env_file:
       - .env
     ports:
       - '127.0.0.1:3000:3000'
       - '127.0.0.1:3001:3001' # Metrics port
     networks:
       - remnawave-network
     depends_on:
       remnawave-db:
         condition: service_healthy
       remnawave-redis:
         condition: service_healthy
     logging:
       driver: 'json-file'
       options:
         max-size: '30m'
         max-file: '5'

7. Restart docker-compose:

   cd /opt/remnawave && docker compose down && docker compose up -d && docker compose logs -f -t

8. Create cadvisor service file:

   nano /etc/systemd/system/cadvisor.service

   Insert the following content:

   cadvisor.service

   [Unit]
   Description=cAdvisor
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=root
   Group=root
   Type=simple
   ExecStart=/opt/monitoring/cadvisor/cadvisor \
           -listen_ip=127.0.0.1 \
           -logtostderr \
           -port=9101 \
           -docker_only=true
   Restart=always
   RestartSec=5
   
   [Install]
   WantedBy=multi-user.target

9. Create node_exporter service file:

   nano /etc/systemd/system/nodeexporter.service

   Insert the following content:

   nodeexporter.service

   [Unit]
   Description=Node Exporter
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=root
   Group=root
   Type=simple
   ExecStart=/opt/monitoring/nodeexporter/node_exporter --web.listen-address=127.0.0.1:9100
   Restart=always
   RestartSec=5
   
   [Install]
   WantedBy=multi-user.target

10. Create vmagent service file:

    nano /etc/systemd/system/vmagent.service

    Attention

    In the line -remoteWrite.url=http://IP:8428/api/v1/write you need to replace with the address of Netbird (in this case the Grafana server)

    Insert the following content:

    vmagent.service

    [Unit]
    Description=VictoriaMetrics Agent
    Wants=network-online.target
    After=network-online.target
    
    [Service]
    User=root
    Group=root
    Type=simple
    ExecStart=/opt/monitoring/vmagent/vmagent \
          -httpListenAddr=127.0.0.1:8429 \
          -promscrape.config=/opt/monitoring/vmagent/scrape.yml \
          -promscrape.configCheckInterval=60s \
          -remoteWrite.url=http://IP:8428/api/v1/write
    Restart=always
    RestartSec=5
    
    [Install]
    WantedBy=multi-user.target

11. Start services and add them to autostart:

    systemctl daemon-reload && systemctl enable cadvisor nodeexporter vmagent && systemctl start cadvisor nodeexporter vmagent

12. Check service status:

    systemctl status cadvisor
    systemctl status nodeexporter
    systemctl status vmagent

Installation and configuration of the server with the Remnawave node

1. Create directories and install cadvisor, node_exporter, vmagent:

   mkdir -p /opt/monitoring/{cadvisor,nodeexporter,vmagent/conf.d} && cd /opt/monitoring/cadvisor && wget https://github.com/google/cadvisor/releases/download/v0.53.0/cadvisor-v0.53.0-linux-amd64 && mv cadvisor-v0.53.0-linux-amd64 cadvisor && chmod +x cadvisor && cd /opt/monitoring/nodeexporter && wget https://github.com/prometheus/node_exporter/releases/download/v1.9.1/node_exporter-1.9.1.linux-amd64.tar.gz && tar -xvf node_exporter-1.9.1.linux-amd64.tar.gz && cd node_exporter-1.9.1.linux-amd64 && mv node_exporter /opt/monitoring/nodeexporter && cd .. && chmod +x node_exporter && rm -f node_exporter-1.9.1.linux-amd64.tar.gz && rm -r node_exporter-1.9.1.linux-amd64 && cd /opt/monitoring/vmagent && wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.123.0/vmutils-linux-amd64-v1.123.0.tar.gz && tar -xvf vmutils-linux-amd64-v1.123.0.tar.gz && mv vmagent-prod vmagent && find . ! -name 'vmagent' -type f -delete && chmod +x vmagent && cd

2. Create vmagent configuration file:

   nano /opt/monitoring/vmagent/scrape.yml

   Insert the following content:

   scrape.yml

   scrape_config_files:
     - "/opt/monitoring/vmagent/conf.d/*.yml"
   
   global:
     scrape_interval: 15s

3. Create cadvisor configuration file:

   nano /opt/monitoring/vmagent/conf.d/cadvisor.yml

   Insert the following content:

   cadvisor.yml

   - job_name: integrations/cAdvisor
     scrape_interval: 15s
     static_configs:
       - targets: ['localhost:9101']
         labels:
           instance: "your_instance_name(server Remnawave node)"

4. Create node_exporter configuration file:

   nano /opt/monitoring/vmagent/conf.d/nodeexporter.yml

   Insert the following content:

   nodeexporter.yml

   - job_name: integrations/node_exporter
     scrape_interval: 15s
     static_configs:
       - targets: ['localhost:9100']
         labels:
           instance: "your_instance_name(server Remnawave node)"

5. Create cadvisor service file:

   nano /etc/systemd/system/cadvisor.service

   Insert the following content:

   cadvisor.service

   [Unit]
   Description=cAdvisor
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=root
   Group=root
   Type=simple
   ExecStart=/opt/monitoring/cadvisor/cadvisor \
           -listen_ip=127.0.0.1 \
           -logtostderr \
           -port=9101 \
           -docker_only=true
   Restart=always
   RestartSec=5
   
   [Install]
   WantedBy=multi-user.target

6. Create node_exporter service file:

   nano /etc/systemd/system/nodeexporter.service

   Insert the following content:

   nodeexporter.service

   [Unit]
   Description=Node Exporter
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=root
   Group=root
   Type=simple
   ExecStart=/opt/monitoring/nodeexporter/node_exporter --web.listen-address=127.0.0.1:9100
   Restart=always
   RestartSec=5
   
   [Install]
   WantedBy=multi-user.target

7. Create vmagent service file:

   nano /etc/systemd/system/vmagent.service

   Attention

   In the line -remoteWrite.url=http://IP:8428/api/v1/write you need to replace with the address of Netbird (in this case the Grafana server)

   Insert the following content:

   vmagent.service

   [Unit]
   Description=VictoriaMetrics Agent
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=root
   Group=root
   Type=simple
   ExecStart=/opt/monitoring/vmagent/vmagent \
         -httpListenAddr=127.0.0.1:8429 \
         -promscrape.config=/opt/monitoring/vmagent/scrape.yml \
         -promscrape.configCheckInterval=60s \
         -remoteWrite.url=http://IP:8428/api/v1/write
   Restart=always
   RestartSec=5
   
   [Install]
   WantedBy=multi-user.target

8. Start services and add them to autostart:

   systemctl daemon-reload && systemctl enable cadvisor nodeexporter vmagent && systemctl start cadvisor nodeexporter vmagent

9. Check service status:

   systemctl status cadvisor
   systemctl status nodeexporter
   systemctl status vmagent

Configure Grafana Dashboards

1. Go to Grafana by the address you specified in nginx.conf

   Grafana URL Example

   https://grafana.yourdomain.com/grafana

2. We meet the authorization page, enter admin and admin, then change the password

3. We get to the main menu of Grafana

   

4. Go to the Connections —> Data Sources section

5. Click the Add data source button and select prometheus

6. In the Connection section, in the URL field, enter http://vmsingle:8428

   

7. Click the Save & Test button at the bottom of the page and check that the connection is established

Adding new Dashboard

List of Dashboard:

• Remnawave Dashboard by kastov
• cAdvisor Dashboard by grafana official
• Node Overview by Kutovoys
• Network System by Kutovoys
• Memory System by Kutovoys
• Filesystem Disks by Kutovoys
• CPU System by Kutovoys

1. Go to the Dashboards section

2. Click the New button on the right and select Import

3. Load our Dashboard

4. Where victoriametrics select our prometheus

   

5. Click the Import button, this template will be loaded