Skip to content

Monitoring with Grafana and Victoria Metrics

Deployment of an intelligent monitoring stack, including Grafana for creating dynamic dashboards, Victoria Metrics for storing and processing metrics, Node Exporter for monitoring servers, cAdvisor for analyzing containers, Xray Checker for controlling network connections, and Netbird for building a secure network architecture. This solution provides maximum transparency, performance, and security for your infrastructure.

The guide and the dashboards come from two authors:

The monitoring guide

A step-by-step guide and the awesome-docs dashboard collection: from the compose file to importing into Grafana.

AuthorpluralplayStoragethe awesome-docs repositoryPanel dashboardby kastov

System dashboards

Five dashboards for watching the servers (Node, Network, Memory, Filesystem, CPU) and co-authorship in the guide.

AuthorkutovoysDashboardsfive ready templatesData sourceVictoria Metrics
  • Grafana - an instrument that transforms various forms of metrics received from Victoria Metrics into visual forms for analytical and system monitoring.

  • Victoria Metrics (VMetrics) - is directly a system for collecting and storing metrics from all machines received from the agent, vmagent - a carrier of these metrics to vmetrics.

  • Node Exporter - an instrument for collecting metrics. It collects information about various resources of the node, such as processor load, memory usage, disk input-output, network statistics, and others.

  • cAdvisor - also an instrument for collecting metrics. It collects information about the performance of Docker containers.

  • Xray-Checker - checks the state of your hosts and sends metrics to Victoria Metrics.

NetBird · WireGuard meshpanel + nodes + monitoring — one private networkpanel servernode_exporter · :9100cAdvisor · :9101panel · :3001node servernode_exporter · :9100cAdvisor · :9101vmagentvmagentremote_writemonitoring VPSVictoria Metricsvmsingle · :8428Xray Checker:2112 · subscriptionGrafana:3000 · dashboardsnginx · :443grafana.yourdomain.combrowser · https

Configuration of your server with Grafana and Victoria Metrics

Section titled “Configuration of your server with Grafana and Victoria Metrics”
  1. Docker installation:

    root@server: ~
    curl -fsSL https://get.docker.com | sh
  2. Create directories and docker-compose.yml:

    root@server: ~
    mkdir -p /opt/monitoring/{cadvisor,nodeexporter,nginx,vmagent/conf.d} && cd /opt/monitoring && nano docker-compose.yml

The web server configuration is your choice; paste your variant into docker-compose.yml:

docker-compose.yml
services:
vmsingle:
image: victoriametrics/victoria-metrics:v1.153.0
container_name: vmsingle
hostname: vmsingle
restart: unless-stopped
volumes:
- ./victoria-metrics-data:/victoria-metrics-data
command:
- '--storageDataPath=/victoria-metrics-data'
networks:
- grafana-network
ports:
- IP:8428:8428 # Here we specify the NetBird address (in this case the Grafana server)
grafana:
image: grafana/grafana:13.2.3
container_name: grafana
hostname: grafana
restart: unless-stopped
volumes:
- grafana-data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning
environment:
- GF_SERVER_DOMAIN=yourdomain.com # The main domain before Grafana, don't forget to add a DNS record for it (if you mounted a subdomain, you need to write the target domain here)
- GF_SERVER_ROOT_URL=https://grafana.yourdomain.com/grafana # Full path to Grafana
- GF_SERVER_SERVE_FROM_SUB_PATH=true
- GF_SERVER_HTTP_PORT=3000
- GF_SERVER_PROTOCOL=http
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=admin
- GF_ANALYTICS_REPORTING_ENABLED=false
networks:
- grafana-network
xray-checker:
image: kutovoys/xray-checker:v1.3.1
container_name: xray-checker
hostname: xraychecker
restart: unless-stopped
environment:
- "SUBSCRIPTION_URL=https://subscription_link" # Here we specify the subscription link for XrayChecker
- "PROXY_STATUS_CHECK_URL=http://google.com/generate_204"
- "PROXY_CHECK_INTERVAL=60"
ports:
- 127.0.0.1:2112:2112
networks:
- grafana-network
grafana-nginx:
image: nginx:1.30.5
container_name: grafana-nginx
volumes:
- ./nginx/nginx.conf:/etc/nginx/conf.d/default.conf:ro
- /etc/letsencrypt/live/yourdomain.com/fullchain.pem:/etc/nginx/ssl/yourdomain.com/fullchain.pem:ro
- /etc/letsencrypt/live/yourdomain.com/privkey.pem:/etc/nginx/ssl/yourdomain.com/privkey.pem:ro
restart: unless-stopped
ports:
- 443:443
networks:
- grafana-network
volumes:
grafana-data:
networks:
grafana-network:
name: grafana-network
driver: bridge
external: false

Create nginx.conf:

root@server: ~
nano /opt/monitoring/nginx/nginx.conf

Insert the following content:

nginx.conf
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ecdh_curve X25519:prime256v1:secp384r1;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;
ssl_prefer_server_ciphers on;
ssl_session_timeout 1d;
ssl_session_cache shared:MozSSL:10m;
ssl_session_tickets off;
server {
listen 443 ssl;
http2 on;
server_name grafana.yourdomain.com; # Write the domain, or subdomain for external access to grafana
ssl_certificate "/etc/nginx/ssl/yourdomain.com/fullchain.pem"; # Path to the certificate
ssl_certificate_key "/etc/nginx/ssl/yourdomain.com/privkey.pem"; # Path to the private key
ssl_trusted_certificate "/etc/nginx/ssl/yourdomain.com/fullchain.pem"; # Path to the certificate
location /grafana {
proxy_pass http://grafana:3000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}

Run docker-compose:

root@server: ~
docker compose up -d && docker compose logs -f -t

Installation and configuration of the Grafana server

Section titled “Installation and configuration of the Grafana server”
  1. Installation of cadvisor, node_exporter, vmagent:

    root@server: ~
    cd /opt/monitoring/cadvisor && wget https://github.com/google/cadvisor/releases/download/v0.60.6/cadvisor-v0.60.6-linux-amd64 && mv cadvisor-v0.60.6-linux-amd64 cadvisor && chmod +x cadvisor && cd /opt/monitoring/nodeexporter && wget https://github.com/prometheus/node_exporter/releases/download/v1.12.1/node_exporter-1.12.1.linux-amd64.tar.gz && tar -xvf node_exporter-1.12.1.linux-amd64.tar.gz && cd node_exporter-1.12.1.linux-amd64 && mv node_exporter /opt/monitoring/nodeexporter && cd .. && chmod +x node_exporter && rm -f node_exporter-1.12.1.linux-amd64.tar.gz && rm -r node_exporter-1.12.1.linux-amd64 && cd /opt/monitoring/vmagent && wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.153.0/vmutils-linux-amd64-v1.153.0.tar.gz && tar -xvf vmutils-linux-amd64-v1.153.0.tar.gz && mv vmagent-prod vmagent && find . ! -name 'vmagent' -type f -delete && chmod +x vmagent && cd
  2. Create vmagent configuration file:

    root@server: ~
    nano /opt/monitoring/vmagent/scrape.yml

    Insert the following content:

    scrape.yml
    scrape_config_files:
    - "/opt/monitoring/vmagent/conf.d/*.yml"
    global:
    scrape_interval: 15s
  3. Create cadvisor configuration file:

    root@server: ~
    nano /opt/monitoring/vmagent/conf.d/cadvisor.yml

    Insert the following content:

    cadvisor.yml
    - job_name: integrations/cAdvisor
    scrape_interval: 15s
    static_configs:
    - targets: ['localhost:9101']
    labels:
    instance: "your_instance_name(server Grafana)"
  4. Create node_exporter configuration file:

    root@server: ~
    nano /opt/monitoring/vmagent/conf.d/nodeexporter.yml

    Insert the following content:

    nodeexporter.yml
    - job_name: integrations/node_exporter
    scrape_interval: 15s
    static_configs:
    - targets: ['localhost:9100']
    labels:
    instance: "your_instance_name(server Grafana)"
  5. Create xray checker configuration file:

    root@server: ~
    nano /opt/monitoring/vmagent/conf.d/xraychecker.yml

    Insert the following content:

    xraychecker.yml
    - job_name: xray_checker
    scrape_interval: 15s
    static_configs:
    - targets: ['127.0.0.1:2112']
    labels:
    instance: "your_instance_name(server Grafana)"
  6. Create cadvisor service file:

    root@server: ~
    nano /etc/systemd/system/cadvisor.service

    Insert the following content:

    cadvisor.service
    [Unit]
    Description=cAdvisor
    Wants=network-online.target
    After=network-online.target
    [Service]
    User=root
    Group=root
    Type=simple
    ExecStart=/opt/monitoring/cadvisor/cadvisor \
    -listen_ip=127.0.0.1 \
    -logtostderr \
    -port=9101 \
    -docker_only=true
    Restart=always
    RestartSec=5
    [Install]
    WantedBy=multi-user.target
  7. Create node_exporter service file:

    root@server: ~
    nano /etc/systemd/system/nodeexporter.service

    Insert the following content:

    nodeexporter.service
    [Unit]
    Description=Node Exporter
    Wants=network-online.target
    After=network-online.target
    [Service]
    User=root
    Group=root
    Type=simple
    ExecStart=/opt/monitoring/nodeexporter/node_exporter --web.listen-address=127.0.0.1:9100
    Restart=always
    RestartSec=5
    [Install]
    WantedBy=multi-user.target
  8. Create vmagent service file:

    root@server: ~
    nano /etc/systemd/system/vmagent.service

    Insert the following content:

    vmagent.service
    [Unit]
    Description=VictoriaMetrics Agent
    Wants=network-online.target
    After=network-online.target
    [Service]
    User=root
    Group=root
    Type=simple
    ExecStart=/opt/monitoring/vmagent/vmagent \
    -httpListenAddr=127.0.0.1:8429 \
    -promscrape.config=/opt/monitoring/vmagent/scrape.yml \
    -promscrape.configCheckInterval=60s \
    -remoteWrite.url=http://IP:8428/api/v1/write
    Restart=always
    RestartSec=5
    [Install]
    WantedBy=multi-user.target
  9. Start services and add them to autostart:

    root@server: ~
    systemctl daemon-reload && systemctl enable cadvisor nodeexporter vmagent && systemctl start cadvisor nodeexporter vmagent
  10. Check service status:

    root@server: ~
    systemctl status cadvisor
    systemctl status nodeexporter
    systemctl status vmagent

Installation and configuration of the server with the Remnawave panel

Section titled “Installation and configuration of the server with the Remnawave panel”
  1. Create directories and install cadvisor, node_exporter, vmagent:

    root@server: ~
    mkdir -p /opt/monitoring/{cadvisor,nodeexporter,vmagent/conf.d} && cd /opt/monitoring/cadvisor && wget https://github.com/google/cadvisor/releases/download/v0.60.6/cadvisor-v0.60.6-linux-amd64 && mv cadvisor-v0.60.6-linux-amd64 cadvisor && chmod +x cadvisor && cd /opt/monitoring/nodeexporter && wget https://github.com/prometheus/node_exporter/releases/download/v1.12.1/node_exporter-1.12.1.linux-amd64.tar.gz && tar -xvf node_exporter-1.12.1.linux-amd64.tar.gz && cd node_exporter-1.12.1.linux-amd64 && mv node_exporter /opt/monitoring/nodeexporter && cd .. && chmod +x node_exporter && rm -f node_exporter-1.12.1.linux-amd64.tar.gz && rm -r node_exporter-1.12.1.linux-amd64 && cd /opt/monitoring/vmagent && wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.153.0/vmutils-linux-amd64-v1.153.0.tar.gz && tar -xvf vmutils-linux-amd64-v1.153.0.tar.gz && mv vmagent-prod vmagent && find . ! -name 'vmagent' -type f -delete && chmod +x vmagent && cd
  2. Create vmagent configuration file:

    root@server: ~
    nano /opt/monitoring/vmagent/scrape.yml

    Insert the following content:

    scrape.yml
    scrape_config_files:
    - "/opt/monitoring/vmagent/conf.d/*.yml"
    global:
    scrape_interval: 15s
  3. Create cadvisor configuration file:

    root@server: ~
    nano /opt/monitoring/vmagent/conf.d/cadvisor.yml

    Insert the following content:

    cadvisor.yml
    - job_name: integrations/cAdvisor
    scrape_interval: 15s
    static_configs:
    - targets: ['localhost:9101']
    labels:
    instance: "your_instance_name(server Remnawave panel)"
  4. Create node_exporter configuration file:

    root@server: ~
    nano /opt/monitoring/vmagent/conf.d/nodeexporter.yml

    Insert the following content:

    nodeexporter.yml
    - job_name: integrations/node_exporter
    scrape_interval: 15s
    static_configs:
    - targets: ['localhost:9100']
    labels:
    instance: "your_instance_name(server Remnawave panel)"
  5. Create remnawave panel configuration file:

    root@server: ~
    nano /opt/monitoring/vmagent/conf.d/remnawave.yml

    Insert the following content:

    Example

    .env
    ### PROMETHEUS ###
    ### Metrics are available at /api/metrics
    METRICS_USER=gjlToiPH
    METRICS_PASS=VsCNutHF
    remnawave.yml
    - job_name: remnawave
    scrape_interval: 15s
    static_configs:
    - targets: ['127.0.0.1:3001']
    labels:
    instance: "your_instance_name(server Remnawave panel)"
    basic_auth:
    username: "xxxxxxx"
    password: "xxxxxxx"
  6. Forward the metrics port in docker-compose.yml:

    root@server: ~
    nano /opt/remnawave/docker-compose.yml

    Find the following content and add:

    docker-compose.yml
    remnawave:
    image: remnawave/backend:latest
    container_name: remnawave
    hostname: remnawave
    restart: always
    env_file:
    - .env
    ports:
    - '127.0.0.1:3000:3000'
    - '127.0.0.1:3001:3001' # Metrics port
    networks:
    - remnawave-network
    depends_on:
    remnawave-db:
    condition: service_healthy
    remnawave-redis:
    condition: service_healthy
    logging:
    driver: 'json-file'
    options:
    max-size: '30m'
    max-file: '5'
  7. Restart docker-compose:

    root@server: ~
    cd /opt/remnawave && docker compose down && docker compose up -d && docker compose logs -f -t
  8. Create cadvisor service file:

    root@server: ~
    nano /etc/systemd/system/cadvisor.service

    Insert the following content:

    cadvisor.service
    [Unit]
    Description=cAdvisor
    Wants=network-online.target
    After=network-online.target
    [Service]
    User=root
    Group=root
    Type=simple
    ExecStart=/opt/monitoring/cadvisor/cadvisor \
    -listen_ip=127.0.0.1 \
    -logtostderr \
    -port=9101 \
    -docker_only=true
    Restart=always
    RestartSec=5
    [Install]
    WantedBy=multi-user.target
  9. Create node_exporter service file:

    root@server: ~
    nano /etc/systemd/system/nodeexporter.service

    Insert the following content:

    nodeexporter.service
    [Unit]
    Description=Node Exporter
    Wants=network-online.target
    After=network-online.target
    [Service]
    User=root
    Group=root
    Type=simple
    ExecStart=/opt/monitoring/nodeexporter/node_exporter --web.listen-address=127.0.0.1:9100
    Restart=always
    RestartSec=5
    [Install]
    WantedBy=multi-user.target
  10. Create vmagent service file:

    root@server: ~
    nano /etc/systemd/system/vmagent.service

    Insert the following content:

    vmagent.service
    [Unit]
    Description=VictoriaMetrics Agent
    Wants=network-online.target
    After=network-online.target
    [Service]
    User=root
    Group=root
    Type=simple
    ExecStart=/opt/monitoring/vmagent/vmagent \
    -httpListenAddr=127.0.0.1:8429 \
    -promscrape.config=/opt/monitoring/vmagent/scrape.yml \
    -promscrape.configCheckInterval=60s \
    -remoteWrite.url=http://IP:8428/api/v1/write
    Restart=always
    RestartSec=5
    [Install]
    WantedBy=multi-user.target
  11. Start services and add them to autostart:

    root@server: ~
    systemctl daemon-reload && systemctl enable cadvisor nodeexporter vmagent && systemctl start cadvisor nodeexporter vmagent
  12. Check service status:

    root@server: ~
    systemctl status cadvisor
    systemctl status nodeexporter
    systemctl status vmagent

Installation and configuration of the server with the Remnawave node

Section titled “Installation and configuration of the server with the Remnawave node”
  1. Create directories and install cadvisor, node_exporter, vmagent:

    root@server: ~
    mkdir -p /opt/monitoring/{cadvisor,nodeexporter,vmagent/conf.d} && cd /opt/monitoring/cadvisor && wget https://github.com/google/cadvisor/releases/download/v0.60.6/cadvisor-v0.60.6-linux-amd64 && mv cadvisor-v0.60.6-linux-amd64 cadvisor && chmod +x cadvisor && cd /opt/monitoring/nodeexporter && wget https://github.com/prometheus/node_exporter/releases/download/v1.12.1/node_exporter-1.12.1.linux-amd64.tar.gz && tar -xvf node_exporter-1.12.1.linux-amd64.tar.gz && cd node_exporter-1.12.1.linux-amd64 && mv node_exporter /opt/monitoring/nodeexporter && cd .. && chmod +x node_exporter && rm -f node_exporter-1.12.1.linux-amd64.tar.gz && rm -r node_exporter-1.12.1.linux-amd64 && cd /opt/monitoring/vmagent && wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.153.0/vmutils-linux-amd64-v1.153.0.tar.gz && tar -xvf vmutils-linux-amd64-v1.153.0.tar.gz && mv vmagent-prod vmagent && find . ! -name 'vmagent' -type f -delete && chmod +x vmagent && cd
  2. Create vmagent configuration file:

    root@server: ~
    nano /opt/monitoring/vmagent/scrape.yml

    Insert the following content:

    scrape.yml
    scrape_config_files:
    - "/opt/monitoring/vmagent/conf.d/*.yml"
    global:
    scrape_interval: 15s
  3. Create cadvisor configuration file:

    root@server: ~
    nano /opt/monitoring/vmagent/conf.d/cadvisor.yml

    Insert the following content:

    cadvisor.yml
    - job_name: integrations/cAdvisor
    scrape_interval: 15s
    static_configs:
    - targets: ['localhost:9101']
    labels:
    instance: "your_instance_name(server Remnawave node)"
  4. Create node_exporter configuration file:

    root@server: ~
    nano /opt/monitoring/vmagent/conf.d/nodeexporter.yml

    Insert the following content:

    nodeexporter.yml
    - job_name: integrations/node_exporter
    scrape_interval: 15s
    static_configs:
    - targets: ['localhost:9100']
    labels:
    instance: "your_instance_name(server Remnawave node)"
  5. Create cadvisor service file:

    root@server: ~
    nano /etc/systemd/system/cadvisor.service

    Insert the following content:

    cadvisor.service
    [Unit]
    Description=cAdvisor
    Wants=network-online.target
    After=network-online.target
    [Service]
    User=root
    Group=root
    Type=simple
    ExecStart=/opt/monitoring/cadvisor/cadvisor \
    -listen_ip=127.0.0.1 \
    -logtostderr \
    -port=9101 \
    -docker_only=true
    Restart=always
    RestartSec=5
    [Install]
    WantedBy=multi-user.target
  6. Create node_exporter service file:

    root@server: ~
    nano /etc/systemd/system/nodeexporter.service

    Insert the following content:

    nodeexporter.service
    [Unit]
    Description=Node Exporter
    Wants=network-online.target
    After=network-online.target
    [Service]
    User=root
    Group=root
    Type=simple
    ExecStart=/opt/monitoring/nodeexporter/node_exporter --web.listen-address=127.0.0.1:9100
    Restart=always
    RestartSec=5
    [Install]
    WantedBy=multi-user.target
  7. Create vmagent service file:

    root@server: ~
    nano /etc/systemd/system/vmagent.service

    Insert the following content:

    vmagent.service
    [Unit]
    Description=VictoriaMetrics Agent
    Wants=network-online.target
    After=network-online.target
    [Service]
    User=root
    Group=root
    Type=simple
    ExecStart=/opt/monitoring/vmagent/vmagent \
    -httpListenAddr=127.0.0.1:8429 \
    -promscrape.config=/opt/monitoring/vmagent/scrape.yml \
    -promscrape.configCheckInterval=60s \
    -remoteWrite.url=http://IP:8428/api/v1/write
    Restart=always
    RestartSec=5
    [Install]
    WantedBy=multi-user.target
  8. Start services and add them to autostart:

    root@server: ~
    systemctl daemon-reload && systemctl enable cadvisor nodeexporter vmagent && systemctl start cadvisor nodeexporter vmagent
  9. Check service status:

    root@server: ~
    systemctl status cadvisor
    systemctl status nodeexporter
    systemctl status vmagent
  1. Go to Grafana by the address you specified in nginx.conf

    Grafana URL Example
    https://grafana.yourdomain.com/grafana
  2. We meet the authorization page, enter admin and admin, then change the password

  3. We get to the main menu of Grafana

    Grafana main menu

  4. Go to the Connections –> Data Sources section

  5. Click the Add data source button and select prometheus

  6. In the Connection section, in the URL field, enter http://vmsingle:8428

    Grafana data source

  7. Click the Save & Test button at the bottom of the page and check that the connection is established

List of Dashboard:

  1. Go to the Dashboards section

  2. Click the New button on the right and select Import

  3. Load our Dashboard

  4. Where victoriametrics select our prometheus

    Grafana Dashboard

  5. Click the Import button, this template will be loaded