G DATA XDR

Reducing Network Traffic Through Update Caching with HAProxy

Introduction

This article describes how to reduce network traffic between the G DATA Agents and the G DATA backend. A locally operated HAProxy caches the actual update file downloads over port 80. The preceding comparison of file lists and hashes is not affected; it is performed independently over port 443 and is not cached.

A caching proxy reduces the volume of data downloaded on the Customer side. Updates are downloaded from the internet only once and then distributed locally to all agents on the network.

This article is primarily intended for administrators with Linux and networking knowledge. Technical terms are also briefly explained so that readers without extensive IT knowledge can follow the procedure.

Background: What Is HAProxy, and What Does a Caching Proxy Do?

When many G DATA Agents run on the same network, each Agent individually downloads security-signature and program updates directly from the G DATA update server over the internet. With one hundred Endpoints, this means one hundred identical downloads over your internet connection.

A proxy is an intermediary between the G DATA Agents and the G DATA backend. A cache is temporary storage. The first request for an update file is forwarded to the internet as usual. The response is also stored locally. Every subsequent request for the same file is served directly from this local storage. This avoids additional internet traffic.

HAProxy is widely used open-source software. It reliably performs this task as a TCP/HTTP proxy with caching functionality.

Prerequisites

  • A Linux server (e.g., Debian/Ubuntu) on the network. All G DATA Agents must be able to reach this server.

  • Root or sudo privileges on this server.

  • A valid TLS certificate, including the certificate chain. It must cover the hostname that agents use to access the proxy via HTTPS (index and metadata requests).

  • Network connectivity permissions:

    • The proxy server must be able to reach the G DATA update server agent-update-server.gdatasecurity.de. This requires two ports: HTTPS port 443 and HTTP port 80.

    • The G DATA Agents must also be able to reach the proxy server over HTTPS port 443 and HTTP port 80.

  • Basic familiarity with HAProxy configuration files is helpful, although this documentation provides a ready-to-use template.

Connectivity to the G DATA Cloud

To ensure reliable and timely reporting of events and Incidents, the G DATA Cloud must be accessible and must not be blocked by a firewall. The settings listed below are required. Ensure that *.gdatasecurity.de and *.gdatasoftware.com can be resolved and are accessible.

  • Port openings

    • The G DATA Agents on the Endpoints must be able to reach our backend server IP addresses via port TCP/443, IP range 194.156.84.0/22 (194.156.84.0 - 194.156.87.255).

  • Protocols

    • The systems must be able to communicate using both HTTPS:// and WSS:// protocols.

  • TLS

  • Peer-to-peer update distribution (Windows systems)

    • To enable peer-to-peer distribution, ports 5353 UDP and 7562 TCP are required during distribution.

Setting Up HAProxy

Step 1: Install HAProxy

Install HAProxy using your Linux distribution’s package manager, for example on Debian/Ubuntu:

sudo apt update
sudo apt install haproxy
Step 2: Prepare the Certificate

HAProxy terminates the agents' client-side TLS connections. To do so, it requires a certificate and its associated private key.

HAProxy expects the certificate, certificate chain, and private key in a single PEM file, combined in this order. The individual components are:

Order Component

1

The certificate itself (server certificate).

2

The intermediate certificates, also called a bundle or chain.

3

The root CA certificate, if available separately.

4

The private key.

Combine the files into a PEM file in exactly this order. Example:

cat server.crt intermediate.crt root-ca.crt server.key > /etc/haproxy/certs/haproxy-cache.pem
chmod 640 /etc/haproxy/certs/haproxy-cache.pem
chown root:haproxy /etc/haproxy/certs/haproxy-cache.pem
Step 3: Create the HAProxy Configuration

Create the configuration file haproxy.cfg in /etc/haproxy/. Use the template below (see Complete Sample Configuration). Replace any existing file with this template.

Adjust the following placeholders:

Placeholder Meaning

{{ haproxy_caching_bind }}

The IP address on which HAProxy listens for connections. This is the IP address the agents use to reach the proxy.

{{ haproxy_caching_server_cert }}

The full path to the PEM certificate file from step 2, e.g., /etc/haproxy/certs/haproxy-cache.pem.

{{ haproxy_caching_server }}

The internal hostname the agents use to access the caching server. This name is expected in the Host header.

Step 4: Start or Restart HAProxy

Restart the service after creating or changing the configuration:

sudo systemctl restart haproxy
sudo systemctl enable haproxy
Step 5 (Optional): Check the Configuration

Check the configuration file’s syntax before restarting HAProxy. This helps identify typos early:

haproxy -c -f /etc/haproxy/haproxy.cfg

A successful check reports Configuration file is valid.

Complete Sample Configuration

The following haproxy.cfg file serves as a template. Replace the three placeholders. Their meanings are explained in Setting Up HAProxy.

global
  log /dev/log  sample 1:10 local0
  log /dev/log  sample 1:10 local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
  stats timeout 30s
  user haproxy
  group haproxy
  daemon
  maxconn                     256000
  hard-stop-after             15m
  tune.ssl.default-dh-param   2048

defaults
  log     global
  mode    http
  option  httplog
  option  dontlognull
  option  http-keep-alive
  option  forwardfor
  timeout     http-request 10s
  timeout     queue 1m
  timeout     connect 10s
  timeout     client 20s
  timeout     server 20s
  timeout     http-keep-alive 10s
  timeout     check 10s
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

frontend stats
  bind {{ haproxy_caching_bind }}:8404 ssl crt {{ haproxy_caching_server_cert }} alpn h2,http/1.1
  stats enable
  stats uri /
  stats refresh 10s
  http-request set-log-level silent
  http-request use-service prometheus-exporter if { path /metrics }

# ---------------------------------------------------------------------------
# Port 80 (HTTP): actual file download - caching takes place HERE.
# ---------------------------------------------------------------------------
frontend http_update_server
  bind {{ haproxy_caching_bind }}:80
  acl is_update_download path_beg /api/v2/update/download/
  acl is_caching_host req.hdr(host) -i {{ haproxy_caching_server }}
  http-request set-header X-Forwarded-Host %[req.hdr(Host)]
  http-request set-header Host %[req.hdr(Host)]
  # Only download requests to the known hostname are served using the cache.
  use_backend http_update_server_backend if is_update_download is_caching_host
  # Redirect all other requests on port 80 to HTTPS as a precaution.
  http-request redirect scheme https code 301 unless is_update_download
  default_backend nomatch

# ---------------------------------------------------------------------------
# Port 443 (HTTPS): index/hash comparison - forward responses WITHOUT caching.
# These responses are small, change frequently, and G DATA deliberately
# does not intend them to be cached.
# ---------------------------------------------------------------------------
frontend https_update_server
  bind {{ haproxy_caching_bind }}:443 ssl crt {{ haproxy_caching_server_cert }} alpn h2,http/1.1
  http-request set-header X-Forwarded-Host %[req.hdr(Host)]
  http-request set-header Host %[req.hdr(Host)]
  use_backend https_update_server_backend if { req.hdr(host) -i {{ haproxy_caching_server }} }
  default_backend nomatch

cache objectcache_update_server
  total-max-size 3000
  max-object-size 200000000
  max-age 86400

backend http_update_server_backend
  balance roundrobin
  http-request set-header Connection keep-alive
  http-request set-header Host agent-update-server.gdatasecurity.de
  http-request set-header X-Forwarded-Proto http
  http-request cache-use objectcache_update_server if { path_beg /api/v2/update/download/ }
  http-response cache-store objectcache_update_server
  option httpchk GET /api/status
  http-check send hdr Host agent-update-server.gdatasecurity.de
  http-check expect status 200
  server agent-update-server.gdatasecurity.de agent-update-server.gdatasecurity.de:80 check inter 10s

backend https_update_server_backend
  balance roundrobin
  http-request set-header Connection keep-alive
  http-request set-header Host agent-update-server.gdatasecurity.de
  http-request set-header X-Forwarded-Proto https
  option httpchk GET /api/status
  http-check send hdr Host agent-update-server.gdatasecurity.de
  http-check expect status 200
  server agent-update-server.gdatasecurity.de agent-update-server.gdatasecurity.de:443 check inter 10s check-sni agent-update-server.gdatasecurity.de sni str(agent-update-server.gdatasecurity.de) ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt

backend nomatch
  errorfile 503 /etc/haproxy/errors/503.http

Configuration Explanation

The key sections are explained individually below so that readers with less HAProxy experience can also understand the configuration.

Section Explanation

global / defaults

Basic process and connection settings. Examples include logging, the maximum number of simultaneous connections (maxconn), and default timeouts. You generally do not need to adjust these values.

frontend stats

Provides a status page on port 8404. This page lets you monitor HAProxy’s status, including in Prometheus format (/metrics).

frontend http_update_server

The entry point for actual file downloads on port 80. Only requests under /api/v2/update/download/ to the known hostname are sent to the caching backend. All other requests on port 80 are redirected to HTTPS as a precaution.

frontend https_update_server

The entry point for index and hash comparisons on port 443 (TLS). Here, HAProxy checks whether the requested hostname ({{ haproxy_caching_server }}) matches the update backend. Only then does HAProxy forward the request. Otherwise, the request is sent to the nomatch backend and receives a 503 error. No caching takes place here.

cache objectcache_update_server

Defines the cache. Three values are important: the maximum total cache size (total-max-size, in MB), the maximum size of an individual file (max-object-size, in bytes), and the maximum time an object remains in the cache (max-age, in seconds). The template uses a total size of approximately 3 GB, up to 200 MB per file, and a validity period of 24 hours. Only the HTTP backend uses this cache.

backend http_update_server_backend

Forwards download requests without encryption (port 80) to agent-update-server.gdatasecurity.de. The line http-request cache-use …​ if { path_beg /api/v2/update/download/ } ensures that only this path is cached. The http-check continuously checks whether the G DATA server is reachable.

backend https_update_server_backend

Forwards index and hash requests to the same G DATA server using encryption (port 443, with certificate validation). This backend contains neither cache-use nor cache-store. Responses are forwarded but not stored.

backend nomatch

Handles all requests that do not match a known host. These requests receive a 503 error. This prevents the configuration from being misused as an open proxy.

HAProxy only caches requests under the path /api/v2/update/download/ over HTTP. Index and hash requests over HTTPS continue unchanged. Other agent communications, such as telemetry or Alerts sent to the XDR backend, are not affected by caching.

Configuring Agents to Use the Caching Server

If you have already installed the G DATA Agents on the Endpoints, contact technical support at +49 234 9762-901. The support team will change your G DATA Agents' configuration for you.

If you have not yet deployed any G DATA Agents, create a setup configuration specifying the proxy address. All Endpoints installed using the associated SetupID will receive their updates through your proxy.

Be sure to verify that the configuration works. Endpoints that do not receive updates pose a security risk. Always keep the fallback server enabled in the configuration (disable it only in justified Exclusion cases). With the fallback server enabled, your Endpoints will also receive updates when they leave the network (for example, laptops) or if your reverse proxy server fails.

Troubleshooting

Symptom Action

HAProxy does not start

Check the configuration with haproxy -c -f /etc/haproxy/haproxy.cfg. Also check the service status with systemctl status haproxy and journalctl -u haproxy.

Agents receive certificate errors

Check two things: Does the PEM file contain the certificate, intermediate certificates, and private key in the correct order? Does the hostname in the certificate match {{ haproxy_caching_server }}?

No noticeable reduction in internet traffic

Use the status page (frontend stats, port 8404) to check whether download requests are being routed through the http_update_server_backend backend and whether cache hits are occurring there. The https_update_server_backend backend intentionally does not cache responses.

Requests are routed to the nomatch backend

The incoming request’s Host header does not match {{ haproxy_caching_server }}. Check the agent configuration and DNS resolution.