Magic WAN and its interoperability
Step 1: Conduit Configuration
Section titled “Step 1: Conduit Configuration”This would be setup during the onboarding with Cloudflare, the setup would require specific information from your end w.r.t specific subnets that should be upgraded or if you want to use non RFC 1918 prefixes.
Step 2: GRE and/or IPsec Tunnel Prerequisites
Section titled “Step 2: GRE and/or IPsec Tunnel Prerequisites”To avoid performance issues caused by packet fragmentation, it’s essential to correctly calculate the Maximum Segment Size (MSS) based on the Maximum Transmission Unit (MTU) and any encapsulation overhead.
-
MTU to MSS Calculation: The fundamental relationship between MTU and MSS for standard TCP/IP traffic is:
-
Standard Internet Values:
- Ethernet MTU: 1500 bytes -> MSS: 1460 bytes
- PPPoE MTU: 1492 bytes (due to 8-byte PPPoE header) -> MSS: 1452 bytes
Generic Routing Encapsulation (GRE) adds a new IP header and a GRE header.
- New IP Header: 20 bytes
- GRE Header: 4 bytes
- Total GRE Overhead: 24 bytes
Example Calculation for GRE over a standard 1500 MTU link:
- Effective Tunnel MTU: bytes
- Optimal MSS: bytes
| Standard Internet Routable MTU | 1500 bytes |
|---|---|
| - Original IP header | 20 bytes |
| - Original protocol header (TCP) | 20 bytes |
| - New IP header | 20 bytes |
| - New protocol header (GRE) | 4 bytes |
| = Maximum segment size (MSS) | 1436 bytes |
The overhead for IPsec’s Encapsulating Security Payload (ESP) is variable and depends on the encryption and integrity algorithms used. It includes a new IP header, ESP header, Initialization Vector (IV), and trailer/authenticator.
A conservative estimate for a common suite like AES-CBC with SHA-256 can be up to 77 bytes. More efficient modern ciphers like AES-GCM have a fixed overhead around 54 bytes.
| Cipher Suite | Mode | New IP Header | ESP Header | IV Size | Padding Range | Trailer | ICV | Total Overhead Range |
|---|---|---|---|---|---|---|---|---|
| AES-256-CBC with SHA-256 | Tunnel | 20 | 8 | 16 | 0-15 | 2 | 16 | 62-77 |
| AES-256-GCM | Tunnel | 20 | 8 | 8 | 0-3 | 2 | 16 | 54-57 |
| 3DES-CBC with SHA-1 | Tunnel | 20 | 8 | 8 | 0-7 | 2 | 12 | 50-57 |
Note: Overhead varies based on padding requirements. AES-CBC uses 16-byte blocks requiring padding to align payload. AES-GCM uses 4-byte alignment. Use the maximum values for conservative MSS calculations.
Example Calculation for IPsec (AES-GCM) over a 1500 MTU link:
- Effective Tunnel MTU: bytes
- Optimal MSS: bytes
MTU/MSS Calculation Cheat Sheet
Section titled “MTU/MSS Calculation Cheat Sheet”This table provides pre-calculated values for common scenarios to serve as a quick reference. Always verify with empirical testing.
| Scenario | Base Internet MTU (Bytes) | Total Overhead (Bytes) | Effective Tunnel MTU (Bytes) | Optimal TCP MSS (Bytes) |
|---|---|---|---|---|
| Standard Internet (DHCP/Static) | 1500 | 40 (IP+TCP) | 1500 | 1460 |
| Standard Internet (PPPoE) | 1492 | 40 (IP+TCP) | 1492 | 1452 |
| GRE Tunnel (over 1500 MTU) | 1500 | 24 | 1476 | 1436 |
| GRE Tunnel (over 1492 MTU) | 1492 | 24 | 1468 | 1428 |
| IPsec Tunnel (AES-GCM, over 1500 MTU) | 1500 | 54 | 1446 | 1406 |
| IPsec Tunnel (AES-CBC/SHA256, over 1500 MTU) | 1500 | 77 (Max) | 1423 | 1383 |
| GRE over IPsec (Transport, AES-GCM, over 1492 MTU) | 1492 | 58 | 1434 | 1394 |
| GRE over IPsec (Tunnel, AES-CBC/SHA256, over 1492 MTU) | 1492 | 97 (Max) | 1395 | 1355 |
MSS Clamping Implementation
Section titled “MSS Clamping Implementation”MSS clamping ensures TCP connections use an appropriate MSS value regardless of what endpoints advertise:
iptables (Linux):
# For GRE tunnelsiptables -t mangle -A FORWARD -o tun0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1436
# For IPsec tunnelsiptables -t mangle -A FORWARD -o vti0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1350
# Auto-clamp to PMTUiptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtunftables (Linux):
table ip mangle { chain forward { type filter hook forward priority -150; oifname "tun0" tcp flags syn tcp option maxseg size set 1436 oifname "vti0" tcp flags syn tcp option maxseg size set 1350 }}Diagnostic Commands and Workflow
Section titled “Diagnostic Commands and Workflow”Do not rely on theoretical values alone. Use the following commands to discover the real-world path limitations and verify your settings.
Step 1: Discover the Path MTU with ping
Section titled “Step 1: Discover the Path MTU with ping”Use an iterative ping test to find the largest packet size that can traverse a path without fragmentation. This is done by setting the “Don’t Fragment” (DF) bit and adjusting the packet size.
Important: The size parameter in ping specifies the ICMP payload. The total packet size is the payload plus 28 bytes (20-byte IP header + 8-byte ICMP header).
Linux Command:
# From behind tunnel, test to remote endpointping -M do -s 1400 <REMOTE_PRIVATE_IP>
# Gradually increase packet size to find maximumfor size in 1400 1420 1440 1460 1472; do echo "Testing size $size:" ping -c 2 -M do -s $size <REMOTE_IP>done
ping <DESTINATION_IP> -M -s 1500-M do: Sets the Don’t Fragment bit.-s <payload_size>: Sets the ICMP payload size in bytes.
Windows Command:
ping <DESTINATION_IP> -f -l <payload_size>-f: Sets the Don’t Fragment bit.-l <payload_size>: Sets the ICMP payload size in bytes.
MacOS Command:
ping -D -s <payload_size> <DESTINATION_IP>-D: Sets the Don’t Fragment bit-s <payload_size>: Sets the ICMP payload size in bytes
Methodology:
- Start with a large payload size, like 1472 (for a 1500-byte packet).
- If you get a “Frag needed and DF set” or “Packet needs to be fragmented” error, the packet is too large.
- Decrease the
<payload_size>by 8 or 10 bytes and repeat until the ping is successful. - The Path MTU is the largest successful
<payload_size>+ 28 bytes. For example, if 1464 is the largest successful payload, your Path MTU is bytes.
Step 2: Verify TCP MSS with tcpdump
Section titled “Step 2: Verify TCP MSS with tcpdump”The MSS value is advertised during the three-way TCP handshake. Use tcpdump to capture the initial SYN packets and inspect the MSS option.[1, 16]
- Command to Capture SYN Packets:
sudo tcpdump -i <INTERFACE> 'tcp[tcpflags] & tcp-syn != 0'sudo tcpdump -i <INTERFACE> 'tcp[tcpflags] & (tcp-syn)!=0 and dst port [443 or 80]This filter is precise, capturing only packets where the SYN flag is set.
Example Output Analysis:
09:47:29.132309 IP [MY_IP].fixed.kpn.net.53958 > 162.159.138.105.https: Flags [S], seq 471050517, win 64240, options [mss 1452,nop,wscale 8,nop,nop,sackOK], length 0With IPsec, 10.68.100.20 is the tunnel endpoint on my side of the tunnel:
09:49:19.268016 IP 10.68.100.20.51678 > 104.16.236.133.https: Flags [S], seq 2995991175, win 32120, options [mss 1350,sackOK,TS val 1792385600 ecr 0,nop,wscale 7], length 0:Step 3: Configure your tunnels and static routes on Cloudflare
Section titled “Step 3: Configure your tunnels and static routes on Cloudflare”- Create a vti with a /31 subnet for use, refer to your vendor documentation on how to create one
- The Cloudflare endpoint will be provided to you via the conduit configuration yaml
- The Customer endpoint would be the IP provided to you via your ISP
- By default, you can only add static routes with RFC 1918 IP prefixes like:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
There are exceptions for publicly routable addresses, inform your friendly (at this point) implementation manager before the project begins.
Step 4: Make sure health-checks work
Section titled “Step 4: Make sure health-checks work”You should start seeing Cloudflare’s side of the tunnel hitting yours (health wise, take an average as not all data centers pinging your end of the tunnel matters):
10:05:07.160315 IP 10.68.100.21 > 10.68.100.20: ICMP echo request, id 7295, seq 0, length 6410:05:07.160378 IP 10.68.100.20 > 10.68.100.21: ICMP echo reply, id 7295, seq 0, length 6410:05:07.169088 IP 10.68.100.21 > 10.68.100.20: ICMP echo request, id 63043, seq 0, length 6410:05:07.169150 IP 10.68.100.20 > 10.68.100.21: ICMP echo reply, id 63043, seq 0, length 6410:05:07.208415 IP 10.68.100.21 > 10.68.100.20: ICMP echo request, id 41057, seq 0, length 6410:05:07.208498 IP 10.68.100.20 > 10.68.100.21: ICMP echo reply, id 41057, seq 0, length 6410:05:07.238198 IP 10.68.100.21 > 10.68.100.20: ICMP echo request, id 17771, seq 0, length 64If you have replay-window enabled, make sure anti-replay protection is enabled on Cloudflare’s side.
Step 5: Route the private subnets to your vti
Section titled “Step 5: Route the private subnets to your vti”Consult your vendor documentation, here are some examples:
- Alibaba Cloud VPN Gateway
- Amazon AWS Transit Gateway
- Aruba EdgeConnect Enterprise
- Cisco IOS XE
- Cisco SD-WAN
- Fortinet
- Furukawa Electric FITELnet
- Google Cloud VPN
- Microsoft Azure
- Palo Alto Networks NGFW
- pfSense
- SonicWall
- Sophos Firewall
- strongSwan
- VyOS
You can do a TCP traceroute to see if it’s going via the tunnel to another endpoint that’s also exposed via Magic WAN:
sudo traceroute -T 172.18.0.8traceroute to 172.18.0.8 (172.18.0.8), 30 hops max, 60 byte packets 1 10.68.69.1 (10.68.69.1) 0.627 ms 0.460 ms 0.480 ms 2 10.68.100.21 (10.68.100.21) 5.833 ms 5.289 ms 5.264 ms 3 172.71.93.32 (172.71.93.32) 7.644 ms 7.038 ms 10.645 ms 4 172.18.0.8 (172.18.0.8) 10.356 ms 10.562 ms 10.244 msStep 6: Automate your provisioning through the use of Gitops/IaC
Section titled “Step 6: Automate your provisioning through the use of Gitops/IaC”This can be done via the UI, API or via Terraform. However, it is recommended to use infrastructure as code as much as possible. The examples make use of .tfvars file and a variables.tf file to reference those sensitive values when applying or planning the infrastructure with Terraform. There are many ways to go about securing the sensitive information including the .tfstate file such as encrypting with Mozilla SOPS and Age.
GRE Tunnels
Section titled “GRE Tunnels”resource "cloudflare_gre_tunnel" "vyos_sg" { account_id = var.cloudflare_account_id name = "vyos_sg" customer_gre_endpoint = var.sg_ip cloudflare_gre_endpoint = var.wan_ip_1 # This will be provided to you during onboarding interface_address = "10.68.88.21/31" description = "vyos_sg_gre" ttl = 64 mtu = 1476 health_check_enabled = true health_check_target = var.sg_ip health_check_type = "request"}IPsec Tunnels
Section titled “IPsec Tunnels”resource "cloudflare_ipsec_tunnel" "vyos_sg_ipsec" { account_id = var.cloudflare_account_id name = "vyos_sg_ipsec" customer_endpoint = var.sg_ip cloudflare_endpoint = var.wan_ip_2 interface_address = "10.68.77.21/31" description = "vyos_sg_ipsec_m1" health_check_enabled = true health_check_target = var.sg_ip health_check_type = "request" psk = var.psk_sg allow_null_cipher = false hex_id = var.hex_id_sg fqdn_id = var.fqdn_id_sg user_id = var.user_id_sg}Static Routes
Section titled “Static Routes”The `next-hop` address would be Cloudflare's side of the tunnelresource "cloudflare_static_route" "eth1_vyos_nl_ipsec" { account_id = var.cloudflare_account_id description = "ETH1" prefix = "10.68.69.0/24" nexthop = "10.68.100.20" priority = 100}resource "cloudflare_static_route" "eth1_100_vyos_nl_ipsec" { account_id = var.cloudflare_account_id description = "VLAN_100" prefix = "10.68.70.0/24" nexthop = "10.68.100.20" priority = 100}
resource "cloudflare_static_route" "podman_vyos_nl_ipsec" { account_id = var.cloudflare_account_id description = "Podman" prefix = "172.18.0.0/16" nexthop = "10.68.100.20" priority = 100}Magic Firewall
Section titled “Magic Firewall”Refer also to example rulesets based on common attack vectors
resource "cloudflare_magic_firewall_ruleset" "magic_firewall" { account_id = var.cloudflare_account_id name = "Magic WAN Firewall" description = "Default Magic WAN Firewall"
rules = [ { action = "allow" expression = "(ip.proto eq \"icmp\")" description = "Allow ICMP" enabled = "true" }, { action = "allow" expression = "(ip.src in {100.64.0.0/10})" description = "Allow WARP Virtual IPs" enabled = "true" } ]}Step 7: Interoperability (Optional)
Section titled “Step 7: Interoperability (Optional)”PAC File (Proxy Endpoints)
Section titled “PAC File (Proxy Endpoints)”There are many ways to deploy the PAC file such as MDMs and using a remote server, in this case, I’ll be using Workers. You can follow the steps here, when using the code, you can create more cases to match specific files, in my case it is nl.pac and sg.pac to simulate the two locations, I also used Workers Secrets to store the Proxy Endpoint domains:
PAC File Worker Deployment
Section titled “PAC File Worker Deployment”Worker entrypoint (index.js)
Section titled “Worker entrypoint (index.js)”import { nl_pac_file, sg_pac_file } from "./pac_file.js";
export default { fetch(request, env) { const url = new URL(request.url);
if (url.pathname === "/nl.pac") { return nl_pac_file(env); } else if (url.pathname === "/sg.pac") { return sg_pac_file(env); } else { return new Response("Not Found", { status: 404 }); } },};PAC File variables (pac_file.js)
Section titled “PAC File variables (pac_file.js)”export function nl_pac_file(env) { const nl = `function FindProxyForURL(url, host) {// No proxy for private (RFC 1918) IP addresses (intranet sites) // if ( // isInNet(dnsResolve(host), "10.0.0.0", "255.0.0.0") || // isInNet(dnsResolve(host), "172.16.0.0", "255.240.0.0") || // isInNet(dnsResolve(host), "192.168.0.0", "255.255.0.0") // ) { // return "DIRECT"; // }
// No proxy for localhost // if (isInNet(dnsResolve(host), "127.0.0.0", "255.0.0.0")) { // return "DIRECT"; // } // Example logic to determine whether to use a proxy return "HTTPS ${env.NL_DOMAIN}.proxy.cloudflare-gateway.com:443";}`; // Set headers to prevent caching const headers = new Headers({ "Content-Type": "application/x-ns-proxy-auto-config", "Cache-Control": "no-store, max-age=0", });
return new Response(nl, { headers: headers });}
export function sg_pac_file(env) { const sg = `function FindProxyForURL(url, host) {// No proxy for private (RFC 1918) IP addresses (intranet sites) // if ( // isInNet(dnsResolve(host), "10.0.0.0", "255.0.0.0") || // isInNet(dnsResolve(host), "172.16.0.0", "255.240.0.0") || // isInNet(dnsResolve(host), "192.168.0.0", "255.255.0.0") // ) { // return "DIRECT"; // }
// No proxy for localhost // if (isInNet(dnsResolve(host), "127.0.0.0", "255.0.0.0")) { // return "DIRECT"; // } // Example logic to determine whether to use a proxy return "HTTPS ${env.SG_DOMAIN}.proxy.cloudflare-gateway.com:443";}`; // Set headers to prevent caching const headers = new Headers({ "Content-Type": "application/x-ns-proxy-auto-config", "Cache-Control": "no-store, max-age=0", });
return new Response(sg, { headers: headers });}wrangler.toml
Section titled “wrangler.toml”name = "worker-proxy-pac"main = "src/index.js"compatibility_date = "2024-05-12"compatibility_flags = ["nodejs_compat"]
[dev]port = 9001local_protocol="http"upstream_protocol="https"
[env.staging]name = "staging-pac"vars = { ENVIRONMENT = "staging" }workers_dev = true
[env.prod]name = "prod-pac"vars = { ENVIRONMENT = "production" }routes = [ { pattern = "proxy.{DOMAIN}.com", custom_domain = true },]# Secrets# NL_DOMAIN# SG_DOMAINGitops/IaC
Section titled “Gitops/IaC”Proxy Endpoints HCL
Section titled “Proxy Endpoints HCL”resource "cloudflare_teams_proxy_endpoint" "nl_proxy_endpoint" { account_id = var.cloudflare_account_id name = "nl" ips = ["${var.nl_ip}/32"]}
resource "cloudflare_teams_proxy_endpoint" "sg_proxy_endpoint" { account_id = var.cloudflare_account_id name = "sg" ips = ["${var.sg_ip}/32"]}WARP/Clientless RBI
Section titled “WARP/Clientless RBI”With WARP, the managed deployment approach is recommended as you can ensure that not only you’re managing the devices and the policies but also the WARP client itself. This can also be managed through the use of Device Profiles. If the use case is to connect to endpoints behind Cloudflare Tunnel or Magic WAN, select the default Virtual Network in the client’s dropdown menu.
Both WARP (you can also isolate applications) and Clientless RBI would use the same Access application and policy as shown below, and both would allow the user to access private IP applications via the other on-ramps.
Gitops/Iac
Section titled “Gitops/Iac”WARP/RBI Access Policy HCL
Section titled “WARP/RBI Access Policy HCL”resource "cloudflare_access_policy" "warp_login" { account_id = var.cloudflare_account_id name = "Allow Erfi" decision = "allow" session_duration = "30m"
include { group = [cloudflare_access_group.erfi_corp.id] }}WARP/RBI Access App HCL
Section titled “WARP/RBI Access App HCL”resource "cloudflare_access_application" "warp_login" { account_id = var.cloudflare_account_id policies = [ cloudflare_access_policy.warp_login.id ] allowed_idps = [ cloudflare_access_identity_provider.entra_id.id, cloudflare_access_identity_provider.google_workspace.id, cloudflare_access_identity_provider.gmail.id, cloudflare_access_identity_provider.keycloak_oidc.id, cloudflare_access_identity_provider.authentik_oidc.id, cloudflare_access_identity_provider.authentik_saml.id, cloudflare_access_identity_provider.otp.id ] auto_redirect_to_identity = false domain = "erfianugrah.cloudflareaccess.com/warp" name = "Warp Login App" session_duration = "24h" type = "warp"}Virtual Networks HCL
Section titled “Virtual Networks HCL”resource "cloudflare_tunnel_virtual_network" "vyos_nl" { account_id = var.cloudflare_account_id name = "vyos_nl_vnet" is_default_network = true}Device Settings Policy HCL
Section titled “Device Settings Policy HCL”resource "cloudflare_device_settings_policy" "default" { account_id = var.cloudflare_account_id name = "default" description = "default_policy" # precedence = 100 # match = "any(identity.groups.name[*] in {\"Erfi Corp\"})" default = true enabled = true allow_mode_switch = true allow_updates = true allowed_to_leave = true auto_connect = 0 captive_portal = 180 disable_auto_fallback = false switch_locked = false service_mode_v2_mode = "warp" service_mode_v2_port = 3000 exclude_office_ips = true}
resource "cloudflare_device_settings_policy" "google" { account_id = var.cloudflare_account_id name = "Google Workspace" description = "google_workspace_policy" precedence = 200 match = "any(identity.groups.name[*] in {\"Erfi Corp\"})" default = false enabled = true allow_mode_switch = true allow_updates = true allowed_to_leave = true auto_connect = 0 captive_portal = 180 disable_auto_fallback = false switch_locked = false service_mode_v2_mode = "warp" service_mode_v2_port = 3000 exclude_office_ips = true}Cloudflare Tunnel
Section titled “Cloudflare Tunnel”Tunnels are used to exposed private applications or for connectivity in the case of RDP, SSH, VNC and the like, it does not support bi-directional traffic as mentioned above. By setting up the routes, you can now reach those same private applications behind Cloudflare Tunnel be it from a PAC file, clientless RBI, WARP or behind Magic WAN.
Gitops/IaC
Section titled “Gitops/IaC”Terraform provider HCL with random provider
Section titled “Terraform provider HCL with random provider”terraform { required_providers { cloudflare = { source = "cloudflare/cloudflare" version = "~> 4.0" } random = { source = "hashicorp/random" version = "~> 3.0" } }}Use the random provider to generate string that will be use to set the tunnel secret
Section titled “Use the random provider to generate string that will be use to set the tunnel secret”# Generate a random stringresource "random_string" "tunnel_secret" { length = 32 special = false}Use the random string as the tunnel secret to create the Cloudflare Tunnel
Section titled “Use the random string as the tunnel secret to create the Cloudflare Tunnel”resource "cloudflare_tunnel" "vyos_nl" { account_id = var.cloudflare_account_id name = "vyos_nl" secret = base64encode(random_string.tunnel_secret.result) config_src = "cloudflare"}Tunnel Route HCL
Section titled “Tunnel Route HCL”resource "cloudflare_tunnel_route" "vyos_nl" { account_id = var.cloudflare_account_id tunnel_id = cloudflare_tunnel.vyos_nl.id network = "0.0.0.0/0" virtual_network_id = cloudflare_tunnel_virtual_network.vyos_nl.id}Tunnel Config HCL
Section titled “Tunnel Config HCL”resource "cloudflare_tunnel_config" "vyos_nl" { account_id = var.cloudflare_account_id tunnel_id = cloudflare_tunnel.vyos_nl.id
config { warp_routing { enabled = true } ingress_rule { hostname = "prom-tunnel-nl.${var.domain_name}" service = "http://localhost:11000" } ingress_rule { hostname = "prom-caddy-nl.${var.domain_name}" service = "http://172.18.0.4:2018" } ingress_rule { service = "http_status:404" } }}(Beta) WARP Connector
Section titled “(Beta) WARP Connector”It shares the same functionality as Cloudflare Tunnel does but with bidirectional support. This means that only you can expose private services/applications which require two-way traffic such as communication services.
As the WARP connector is also essentially a device connected via WARP to the internet, a remote user with WARP connectivity can also connect to it directly and not just the services exposed behind it which is what we call WARP-to-WARP connectivity. Both site-to-site and peer-to-peer use cases work here, similar to Magic WAN.
Site-to-site Connectivity
Section titled “Site-to-site Connectivity”- Setup Access policy for WARP enrolment (this would be the same policy for WARP authentication), since it’s site-to-site, we will have to deploy it via
.xmlwith the required parameters.
Refer to service tokens and device enrolment for details.
- Setup the
split tunnelto eitherincludeorexcludemode, in this example, I have chosenexcludeso that I can still SSH into the nodes to showcase the functionality.
Since this portion can be managed by Terraform, this is the example HCL:
resource "cloudflare_split_tunnel" "default_include" { account_id = var.cloudflare_account_id policy_id = cloudflare_device_settings_policy.default.id mode = "exclude" tunnels { address = "10.68.69.3" description = "ERFI1" } tunnels { address = "10.68.73.3" description = "arch-0" } tunnels { address = "10.68.73.2" description = "pve" }}- Ensure these settings are enabled:
- Enable CGNAT routing (Settings -> Network)
- Enable Proxy (UDP, TCP, ICMP)
- Enable WARP to WARP
- Override local interface IP (Settings -> WARP Client)
- If in
includemode, ensure100.96.0.0/12is included
- If in
Some of these settings are available as a Terraform resource (as shown below), since it would be a mix a of API/UI and Terraform, Terraform would not know the state, so just keep to API/UI for all settings.
resource "cloudflare_teams_account" "miau3" { account_id = var.cloudflare_account_id tls_decrypt_enabled = false protocol_detection_enabled = false activity_log_enabled = true non_identity_browser_isolation_enabled = true
body_scanning { inspection_mode = "deep" }
antivirus { enabled_download_phase = false enabled_upload_phase = false fail_closed = false }
fips { tls = false }
proxy { tcp = true udp = true root_ca = true }
url_browser_isolation_enabled = true
logging { redact_pii = false settings_by_rule_type { dns { log_all = true log_blocks = false } http { log_all = true log_blocks = false } l4 { log_all = true log_blocks = false } } }
extended_email_matching { enabled = true }}- Set up the WARP Connector
- Either via API or the UI, create the WARP connector (currently not available as a Terraform resource):
curl --request POST \ --url https://api.cloudflare.com/client/v4/accounts/account_id/warp_connector \ --header 'Content-Type: application/json' \ --header 'X-Auth-Email: ' \ --data '{ "name": "arch-0"}'Upon creation, you will get the example .xml e.g. which will look like this:
<dict> <key>organization</key> <string>org_name</string> <key>auth_client_id</key> <string>client_access_id_string</string> <key>auth_client_secret</key> <string>client_access_secret_string</string> <key>warp_connector_token</key> <string>connector_token_string</string></dict>This is what you will place in the /var/lib/cloudflare-warp directory after the WARP client install. Refer to the relevant repositories that matches your Linux distribution, for Arch, you would have to install via AUR.
Once done, if not yet enabled, run sudo systemctl enable warp-svc then sudo systemctl start warp-svc, if already running as a service, run sudo systemctl restart warp-svc so that it will use the new configuration to connect to Cloudflare.
- Set up Linux networking
We need to enable IP forwarding and MSS clamping (WARP’s MTU is 1280 bytes).
Create a .conf file in the etc/sysctl.d/ directory with value of net.ipv4.ip_forward = 1 so that this persists upon reboot.
To check what IPtables rules are existing, we can run sudo iptables-save > test.conf to just see what’s currently being use, if all’s well. We can add the following to the existing /etc/iptables/iptables.rules file, such that it would look something like this:
*filter:INPUT ACCEPT [0:0]:FORWARD ACCEPT [0:0]:OUTPUT ACCEPT [0:0]-A FORWARD -i CloudflareWARP -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu-A FORWARD -o CloudflareWARP -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtuCOMMITThis would make sure the MSS clamping rules will also persist on reboot.
- Route nodes/VMs/containers to the WARP connector
The most common use case here would either be an alternate gateway to the main router or an intermediate gateway. In this example, I will be using it as an intermediate gateway.
Set up the static routes for each WARP connector, the configuration will be similar to the Cloudflare Tunnel routes. In our case, on the WARP connector called arch-1, we will exposing 10.68.73.102/32 for the arch-2 machine, and arch-3 will be exposing 10.68.73.104/32 for arch-4 machine.
Let’s use arch-2 as an example as the same steps will be replicated on the other node.
Run:
ip routeand you should be able to see default routes, for instance:
default via 10.68.73.1 dev ens18 proto dhcp src 10.68.73.102 metric 100we want to create another route that has higher precedence than this (lower number).
Run:
sudo ip route add default via 10.68.73.101 dev ens18 metric 99What this means is that, traffic from arch-2 will go via the ens18 interface to 10.68.73.101 which is the WARP connector arch-1 and using it as a gateway, now this node is connected to Cloudflare.
Do the same for the other node, and change the IP.
Run:
curl https://icanhazip.comon either machine, and you should be able to see a Cloudflare IP, just be aware since all traffic is sent to the WARP connector, that the DNS server used by the machines routed to WARP connector can be accessed while on the network, either by another route or a public DNS server.
curl icanhazip.com104.28.218.39- Route the machines to each other
We can either router the entire prefix to the WARP connector or specific IPs, in my case, I will be using just one IP:
sudo ip route add 10.68.73.104 via 10.68.73.101 dev ens18 metric 98This will allow 10.68.73.102 to reach 10.68.73.104, do the same on arch-4, just by swapping the IPs, from .104 to .102
Run:
ping 10.68.73.102or
mtr 10.68.73.102and vice versa to see the traffic reach the WARP connector CGNAT IP before reaching the other machine.
Footnotes
Section titled “Footnotes”-
The arrows signify bidirectional connections or unidirectional ↩