Monitoring with Prometheus and Grafana

Author: OMchik33
A complete guide to setting up monitoring for the Remnawave panel using Prometheus, Grafana, and Node Exporter, as well as secure access to metrics via an SSH tunnel and Nginx.

Important

This guide is “lite” version of setting up grafana on your server and describes the installation of services on a separate monitoring VPS server. The steps are described for a root user.

📦 Setting up docker-compose.yml on the Remnawave server

Make sure that port 3001 is open in the Remnawave docker-compose.yml file, through which metrics are provided:

ports:
  - '127.0.0.1:3001:3001'

Example configuration (for installation using the eGames script)

remnawave:
  image: remnawave/backend:latest
  container_name: remnawave
  hostname: remnawave
  restart: always
  env_file:
    - .env
  ports:
    - '127.0.0.1:3000:3000'
    - '127.0.0.1:3001:3001'
  networks:
    - remnawave-network
  depends_on:
    remnawave-db:
      condition: service_healthy
    remnawave-redis:
      condition: service_healthy
  logging:
    driver: 'json-file'
    options:
      max-size: '30m'
      max-file: '5'

🔐 Setting up an SSH tunnel

Note

A tunnel is needed if metrics are protected by an authorization cookie, for example, when installing Remnawave using the eGames script.

Caution

To be executed on the monitoring server

1. Generate a key, do not set a password:

   ssh-keygen -t ed25519 -f ~/.ssh/remna_tunnel_key

2. Add the public key to the ~/.ssh/authorized_keys file on the Remnawave server. Replace AAAAC3... with the content of remna_tunnel_key.pub, which starts with AAAAC3:

   from="MONITORING_SERVER_IP",no-pty,no-agent-forwarding,no-X11-forwarding,command="/bin/false" ssh-ed25519 AAAAC3... remna_tunnel_key

3. Install autossh (continue on the monitoring server):

   sudo apt install autossh

4. Create a systemd service file at /etc/systemd/system/remna-tunnel.service. Replace remnauser@REMNA_SERVER_IP with the SSH login and address of the Remnawave panel server.

   [Unit]
   Description=SSH tunnel to Remnawave for Prometheus and Node Exporter
   After=network.target
   
   [Service]
   User=root
   Environment="AUTOSSH_GATETIME=0"
   ExecStart=/usr/bin/autossh -N \
   -o "ServerAliveInterval 60" \
   -o "ServerAliveCountMax 3" \
   -i /root/.ssh/remna_tunnel_key \
   -L 9001:localhost:3001 \
   -L 9002:localhost:9100 \
   remnauser@REMNA_SERVER_IP
   Restart=always
   
   [Install]
   WantedBy=multi-user.target

5. On the first server (where autossh is running), it’s important to add the second server’s key to known_hosts, where 12.345.678.91 is the IP address of the Remnawave panel, to avoid a host authenticity verification error. Otherwise, the service will simply not connect to the panel server.

   ssh-keyscan -H 12.345.678.91 >> ~/.ssh/known_hosts

6. Start the service:

   sudo systemctl daemon-reexec
   sudo systemctl enable remna-tunnel
   sudo systemctl start remna-tunnel

7. Check if the service started successfully:

   sudo systemctl status remna-tunnel

Now Remnawave and Node Exporter metrics are available at http://localhost:9001/metrics and http://localhost:9002/metrics on the monitoring server.

📈 Installing Prometheus and Grafana

1. Create directories:

   mkdir -p /opt/monitoring/{grafana,prometheus}

2. Create the file /opt/monitoring/docker-compose.yml

   services:
     prometheus:
       image: prom/prometheus:latest
       container_name: prometheus
       restart: unless-stopped
       volumes:
         - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
         - prometheus-data:/prometheus
       command:
         - '--config.file=/etc/prometheus/prometheus.yml'
         - '--storage.tsdb.path=/prometheus'
         - '--web.console.libraries=/etc/prometheus/console_libraries'
         - '--web.console.templates=/etc/prometheus/consoles'
         - '--web.external-url=https://sub.mydomain.com/prometheus/'
         - '--web.route-prefix=/'
       network_mode: host
   
     grafana:
       image: grafana/grafana:latest
       container_name: grafana
       restart: unless-stopped
       volumes:
         - grafana-data:/var/lib/grafana
         - ./grafana/provisioning:/etc/grafana/provisioning
       environment:
         - GF_SERVER_DOMAIN=mydomain.com
         - GF_SERVER_ROOT_URL=https://sub.mydomain.com/grafana
         - GF_SERVER_SERVE_FROM_SUB_PATH=true
         - GF_SERVER_HTTP_PORT=3000
         - GF_SERVER_PROTOCOL=http
         - GF_SECURITY_ADMIN_USER=admin
         - GF_SECURITY_ADMIN_PASSWORD=admin
         - GF_ANALYTICS_REPORTING_ENABLED=false
       network_mode: host
   
     xray-checker:
       image: kutovoys/xray-checker
       environment:
         - "SUBSCRIPTION_URL=https://podpiska.mydomain.com/6f5g46df46g45f54"
         - "PROXY_STATUS_CHECK_URL=http://google.com/generate_204"
         - "PROXY_CHECK_INTERVAL=60"
       network_mode: host
   
   volumes:
     prometheus-data:
     grafana-data:

Here sub.mydomain.com is the domain address attached to the monitoring VPS where Grafana and Prometheus are installed.

https://podpiska.mydomain.com/6f5g46df46g45f54 is the subscription; create a separate user for this role.

⚙️ Configuring Prometheus

1. File /opt/monitoring/prometheus/prometheus.yml. username and password are from the Remnawave .env file (section ### PROMETHEUS ###)

   global:
     scrape_interval: 15s
   
   scrape_configs:
     # For proper operation of the Grafana dashboards listed in the links below,
     # use job_name: integrations/node_exporter.
     - job_name: 'node_exporter'
       static_configs:
         - targets: ['127.0.0.1:9002']
           labels:
             cluster: "test"
             instance: "127.0.0.1:9002"
     - job_name: 'remnawave_exporter'
       static_configs:
         - targets: ['127.0.0.1:9001']
       basic_auth:
         username: "XXXXXXXXXXXXXXX"
         password: "XXXXXXXXXXXXXXX"
     - job_name: "xray-checker"
       metrics_path: "/metrics"
       static_configs:
         - targets: ["localhost:2112"]
       scrape_interval: 1m

2. Start all containers:

   cd /opt/monitoring && docker compose up -d

🌐 Setting up Nginx and SSL

1. Install nginx (local installation, not in a Docker container):

   apt install nginx

2. Obtain SSL certificates; the 3rd level domain should already be configured to the IP address of the monitoring server:

   sudo apt install certbot python3-certbot-nginx
   sudo certbot --nginx -d sub.mydomain.com

3. Add auto-renewal of certificates to crontab:

   0 5 * * * root certbot renew --quiet

4. Configure the nginx config file

   nano /etc/nginx/sites-available/default

Example nginx configuration

# Check by cookie
map $http_cookie $auth_cookie {
    default 0;
    "~*fd4gd54fg2dfg4241=1" 1;
}

# Check by GET parameter
map $arg_fd4gd54fg2dfg4241 $auth_query {
    default 0;
    "1" 1;
}

# General authorization flag
map "$auth_cookie$auth_query" $authorized {
    "~1" 1;
    default 0;
}

# Set cookie if parameter exists
map $arg_fd4gd54fg2dfg4241 $set_cookie_header {
    "1" "fd4gd54fg2dfg4241=1; Path=/; HttpOnly; Secure; SameSite=Strict; Max-Age=31536000";
    default "";
}

# HTTP redirect to HTTPS
server {
    listen 80;
    server_name sub.mydomain.com;
    return 301 https://$server_name$request_uri;
}

# HTTPS server block
server {
    listen 443 ssl http2;
    server_name sub.mydomain.com;

    # SSL configuration
    ssl_certificate /etc/letsencrypt/live/sub.mydomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/sub.mydomain.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;

    # Set the cookie if the user logs in via a link
    add_header Set-Cookie $set_cookie_header;

    # Redirect from the main domain directly to the desired public dashboard in Grafana
    location = / {
        return 301 /grafana/public-dashboards/f5g4df4g5df4gd5f4g63d4834379e;
    }

    # Grafana configuration
    location /grafana {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $server_name;

        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        # Remove Authorization header
        proxy_set_header Authorization "";
    }

    # Grafana Live WebSocket
    location /grafana/api/live/ {
        proxy_pass http://localhost:3000/api/live/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Prometheus
    location /prometheus/ {
        if ($authorized = 0) {
            return 404;
        }

        proxy_pass http://localhost:9090/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $server_name;
        proxy_set_header Authorization "";
    }

    # Xray Checker
    location /checker/ {
        if ($authorized = 0) {
            return 404;
        }

        proxy_pass http://localhost:2112/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $server_name;
    }
}

Here sub.mydomain.com is the domain address attached to the test VPS where Grafana and Prometheus are installed.

fd4gd54fg2dfg4241 is a unique cookie value, YOU NEED to replace it with your own.

/grafana/public-dashboards/f5g4df4g5df4gd5f4g63d4834379e; is a unique link to a public Dashboard in Grafana. You will get it after completing all the instructions and building the necessary Dashboard yourself. After that, you can get the link for external connection, insert it here and restart nginx. The link in the nginx config is set so that users are immediately redirected to the public Dashboard when they go to sub.mydomain.com.

1. Edit /etc/nginx/nginx.conf, add there:

           # Map for Grafana WebSocket connections
           map $http_upgrade $connection_upgrade {
           default upgrade;
           '' close;
           }

2. Apply nginx configuration

   nginx -t && systemctl restart nginx

Example of a default nginx.conf with the necessary map added

Open an example

user www-data;
worker_processes auto;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        # Map for Grafana WebSocket connections
        map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
        }
        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;

        ##
        # Gzip Settings
        ##

        gzip on;

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/jav>

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

🧠 Node Exporter

1. Install on the server with Remnawave:

   . <(wget -qO- https://raw.githubusercontent.com/g7AzaZLO/NodeExporter-autoinstaller/main/NodeExporter-auto-install.sh)

2. You can install it on other nodes and add it to prometheus.yml:

   - job_name: 'external_nodes'
     static_configs:
       - targets: ['1.2.3.4:9100']

Node Exporter is available on the monitoring server at localhost:9002 (via SSH tunnel).

Here 1.2.3.4 is the address of another node where we also installed Node Exporter.

Or use SSH tunnels similar to this guide.

📊 Checking and configuring Grafana

Authentication in Grafana

• Go to: https://sub.mydomain.com/grafana
• Login: admin / admin, then change the password
• Add a data source: Prometheus
  • URL: http://localhost:9090
• Go to Explore → Metrics → Grafana Drilldown → Metrics

After all the manipulations, we have added a new data source to Grafana - Prometheus. Now you need to create a new Dashboard yourself, adding the necessary visualizations and connecting the required data sources from Prometheus to them.

Checking the health of data sources (metrics) in Prometheus

1. Go to the link https://sub.mydomain.com/prometheus/graph?fd4gd54fg2dfg4241=1

2. Open the Status - Target Health menu and make sure that each data source is UP.

3. Open Prometheus https://sub.mydomain.com/prometheus/graph?fd4gd54fg2dfg4241=1

4. Open Grafana https://sub.mydomain.com/grafana/

5. Open xray checker https://sub.mydomain.com/checker/?fd4gd54fg2dfg4241=1

Note

If you changed the cookie code in the nginx settings, you need to change it in the links above as well.

Tip

You can download an example visualization of node exporter

• Dashboard ID: 1860
• https://grafana.com/grafana/dashboards/1860

📙 Useful links

• An example of metrics