G DATA XDR
Reducing Network Traffic Through Update Caching with HAProxy
Introduction
This article describes how to reduce network traffic between the G DATA Agents and the G DATA backend. A locally operated HAProxy caches the actual update file downloads over port 80. The preceding comparison of file lists and hashes is not affected; it is performed independently over port 443 and is not cached.
A caching proxy reduces the volume of data downloaded on the Customer side. Updates are downloaded from the internet only once and then distributed locally to all agents on the network.
|
This article is primarily intended for administrators with Linux and networking knowledge. Technical terms are also briefly explained so that readers without extensive IT knowledge can follow the procedure. |
Background: What Is HAProxy, and What Does a Caching Proxy Do?
When many G DATA Agents run on the same network, each Agent individually downloads security-signature and program updates directly from the G DATA update server over the internet. With one hundred Endpoints, this means one hundred identical downloads over your internet connection.
A proxy is an intermediary between the G DATA Agents and the G DATA backend. A cache is temporary storage. The first request for an update file is forwarded to the internet as usual. The response is also stored locally. Every subsequent request for the same file is served directly from this local storage. This avoids additional internet traffic.
HAProxy is widely used open-source software. It reliably performs this task as a TCP/HTTP proxy with caching functionality.
Prerequisites
-
A Linux server (e.g., Debian/Ubuntu) on the network. All G DATA Agents must be able to reach this server.
-
Root or sudo privileges on this server.
-
A valid TLS certificate, including the certificate chain. It must cover the hostname that agents use to access the proxy via HTTPS (index and metadata requests).
-
Network connectivity permissions:
-
The proxy server must be able to reach the G DATA update server agent-update-server.gdatasecurity.de. This requires two ports: HTTPS port 443 and HTTP port 80.
-
The G DATA Agents must also be able to reach the proxy server over HTTPS port 443 and HTTP port 80.
-
-
Basic familiarity with HAProxy configuration files is helpful, although this documentation provides a ready-to-use template.
Connectivity to the G DATA Cloud
To ensure reliable and timely reporting of events and Incidents, the G DATA Cloud must be accessible and must not be blocked by a firewall. The settings listed below are required. Ensure that *.gdatasecurity.de and *.gdatasoftware.com can be resolved and are accessible.
-
Port openings
-
The G DATA Agents on the Endpoints must be able to reach our backend server IP addresses via port TCP/443, IP range 194.156.84.0/22 (194.156.84.0 - 194.156.87.255).
-
-
Protocols
-
The systems must be able to communicate using both HTTPS:// and WSS:// protocols.
-
-
TLS
-
Proxy servers in use must support at least TLS 1.2.
-
Neither SSL Inspection nor Deep Packet Inspection may be used.
-
It must be possible to establish a TLS-encrypted connection to chess.gdatasecurity.de. An appropriate root certificate must be available for this purpose.
-
-
Peer-to-peer update distribution (Windows systems)
-
To enable peer-to-peer distribution, ports 5353 UDP and 7562 TCP are required during distribution.
-
Setting Up HAProxy
Step 1: Install HAProxy
Install HAProxy using your Linux distribution’s package manager, for example on Debian/Ubuntu:
sudo apt update
sudo apt install haproxy
Step 2: Prepare the Certificate
HAProxy terminates the agents' client-side TLS connections. To do so, it requires a certificate and its associated private key.
|
HAProxy expects the certificate, certificate chain, and private key in a single PEM file, combined in this order. The individual components are: |
| Order | Component |
|---|---|
1 |
The certificate itself (server certificate). |
2 |
The intermediate certificates, also called a bundle or chain. |
3 |
The root CA certificate, if available separately. |
4 |
The private key. |
Combine the files into a PEM file in exactly this order. Example:
cat server.crt intermediate.crt root-ca.crt server.key > /etc/haproxy/certs/haproxy-cache.pem
chmod 640 /etc/haproxy/certs/haproxy-cache.pem
chown root:haproxy /etc/haproxy/certs/haproxy-cache.pem
Step 3: Create the HAProxy Configuration
Create the configuration file haproxy.cfg in /etc/haproxy/. Use the template below (see Complete Sample Configuration). Replace any existing file with this template.
Adjust the following placeholders:
| Placeholder | Meaning |
|---|---|
{{ haproxy_caching_bind }} |
The IP address on which HAProxy listens for connections. This is the IP address the agents use to reach the proxy. |
{{ haproxy_caching_server_cert }} |
The full path to the PEM certificate file from step 2, e.g., /etc/haproxy/certs/haproxy-cache.pem. |
{{ haproxy_caching_server }} |
The internal hostname the agents use to access the caching server. This name is expected in the Host header. |
Step 4: Start or Restart HAProxy
Restart the service after creating or changing the configuration:
sudo systemctl restart haproxy
sudo systemctl enable haproxy
Step 5 (Optional): Check the Configuration
Check the configuration file’s syntax before restarting HAProxy. This helps identify typos early:
haproxy -c -f /etc/haproxy/haproxy.cfg
A successful check reports Configuration file is valid.
Complete Sample Configuration
The following haproxy.cfg file serves as a template. Replace the three placeholders. Their meanings are explained in Setting Up HAProxy.
global
log /dev/log sample 1:10 local0
log /dev/log sample 1:10 local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
maxconn 256000
hard-stop-after 15m
tune.ssl.default-dh-param 2048
defaults
log global
mode http
option httplog
option dontlognull
option http-keep-alive
option forwardfor
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 20s
timeout server 20s
timeout http-keep-alive 10s
timeout check 10s
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend stats
bind {{ haproxy_caching_bind }}:8404 ssl crt {{ haproxy_caching_server_cert }} alpn h2,http/1.1
stats enable
stats uri /
stats refresh 10s
http-request set-log-level silent
http-request use-service prometheus-exporter if { path /metrics }
# ---------------------------------------------------------------------------
# Port 80 (HTTP): actual file download - caching takes place HERE.
# ---------------------------------------------------------------------------
frontend http_update_server
bind {{ haproxy_caching_bind }}:80
acl is_update_download path_beg /api/v2/update/download/
acl is_caching_host req.hdr(host) -i {{ haproxy_caching_server }}
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
http-request set-header Host %[req.hdr(Host)]
# Only download requests to the known hostname are served using the cache.
use_backend http_update_server_backend if is_update_download is_caching_host
# Redirect all other requests on port 80 to HTTPS as a precaution.
http-request redirect scheme https code 301 unless is_update_download
default_backend nomatch
# ---------------------------------------------------------------------------
# Port 443 (HTTPS): index/hash comparison - forward responses WITHOUT caching.
# These responses are small, change frequently, and G DATA deliberately
# does not intend them to be cached.
# ---------------------------------------------------------------------------
frontend https_update_server
bind {{ haproxy_caching_bind }}:443 ssl crt {{ haproxy_caching_server_cert }} alpn h2,http/1.1
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
http-request set-header Host %[req.hdr(Host)]
use_backend https_update_server_backend if { req.hdr(host) -i {{ haproxy_caching_server }} }
default_backend nomatch
cache objectcache_update_server
total-max-size 3000
max-object-size 200000000
max-age 86400
backend http_update_server_backend
balance roundrobin
http-request set-header Connection keep-alive
http-request set-header Host agent-update-server.gdatasecurity.de
http-request set-header X-Forwarded-Proto http
http-request cache-use objectcache_update_server if { path_beg /api/v2/update/download/ }
http-response cache-store objectcache_update_server
option httpchk GET /api/status
http-check send hdr Host agent-update-server.gdatasecurity.de
http-check expect status 200
server agent-update-server.gdatasecurity.de agent-update-server.gdatasecurity.de:80 check inter 10s
backend https_update_server_backend
balance roundrobin
http-request set-header Connection keep-alive
http-request set-header Host agent-update-server.gdatasecurity.de
http-request set-header X-Forwarded-Proto https
option httpchk GET /api/status
http-check send hdr Host agent-update-server.gdatasecurity.de
http-check expect status 200
server agent-update-server.gdatasecurity.de agent-update-server.gdatasecurity.de:443 check inter 10s check-sni agent-update-server.gdatasecurity.de sni str(agent-update-server.gdatasecurity.de) ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt
backend nomatch
errorfile 503 /etc/haproxy/errors/503.http
Configuration Explanation
The key sections are explained individually below so that readers with less HAProxy experience can also understand the configuration.
| Section | Explanation |
|---|---|
global / defaults |
Basic process and connection settings. Examples include logging, the maximum number of simultaneous connections (maxconn), and default timeouts. You generally do not need to adjust these values. |
frontend stats |
Provides a status page on port 8404. This page lets you monitor HAProxy’s status, including in Prometheus format (/metrics). |
frontend http_update_server |
The entry point for actual file downloads on port 80. Only requests under /api/v2/update/download/ to the known hostname are sent to the caching backend. All other requests on port 80 are redirected to HTTPS as a precaution. |
frontend https_update_server |
The entry point for index and hash comparisons on port 443 (TLS). Here, HAProxy checks whether the requested hostname ({{ haproxy_caching_server }}) matches the update backend. Only then does HAProxy forward the request. Otherwise, the request is sent to the nomatch backend and receives a 503 error. No caching takes place here. |
cache objectcache_update_server |
Defines the cache. Three values are important: the maximum total cache size (total-max-size, in MB), the maximum size of an individual file (max-object-size, in bytes), and the maximum time an object remains in the cache (max-age, in seconds). The template uses a total size of approximately 3 GB, up to 200 MB per file, and a validity period of 24 hours. Only the HTTP backend uses this cache. |
backend http_update_server_backend |
Forwards download requests without encryption (port 80) to agent-update-server.gdatasecurity.de. The line http-request cache-use … if { path_beg /api/v2/update/download/ } ensures that only this path is cached. The http-check continuously checks whether the G DATA server is reachable. |
backend https_update_server_backend |
Forwards index and hash requests to the same G DATA server using encryption (port 443, with certificate validation). This backend contains neither cache-use nor cache-store. Responses are forwarded but not stored. |
backend nomatch |
Handles all requests that do not match a known host. These requests receive a 503 error. This prevents the configuration from being misused as an open proxy. |
|
HAProxy only caches requests under the path /api/v2/update/download/ over HTTP. Index and hash requests over HTTPS continue unchanged. Other agent communications, such as telemetry or Alerts sent to the XDR backend, are not affected by caching. |
Configuring Agents to Use the Caching Server
If you have already installed the G DATA Agents on the Endpoints, contact technical support at +49 234 9762-901. The support team will change your G DATA Agents' configuration for you.
If you have not yet deployed any G DATA Agents, create a setup configuration specifying the proxy address. All Endpoints installed using the associated SetupID will receive their updates through your proxy.
|
Be sure to verify that the configuration works. Endpoints that do not receive updates pose a security risk. Always keep the fallback server enabled in the configuration (disable it only in justified Exclusion cases). With the fallback server enabled, your Endpoints will also receive updates when they leave the network (for example, laptops) or if your reverse proxy server fails. |
Troubleshooting
| Symptom | Action |
|---|---|
HAProxy does not start |
Check the configuration with haproxy -c -f /etc/haproxy/haproxy.cfg. Also check the service status with systemctl status haproxy and journalctl -u haproxy. |
Agents receive certificate errors |
Check two things: Does the PEM file contain the certificate, intermediate certificates, and private key in the correct order? Does the hostname in the certificate match {{ haproxy_caching_server }}? |
No noticeable reduction in internet traffic |
Use the status page (frontend stats, port 8404) to check whether download requests are being routed through the http_update_server_backend backend and whether cache hits are occurring there. The https_update_server_backend backend intentionally does not cache responses. |
Requests are routed to the nomatch backend |
The incoming request’s Host header does not match {{ haproxy_caching_server }}. Check the agent configuration and DNS resolution. |