Caddy Reverse Proxy¶
Imported from
smartwatermelon/tilsit-caddy(archived 2026-04-03). For full git history, see the archived repository.
This file provides operational reference for the Caddy reverse proxy, dashboard, and media file server components.
Project Overview¶
Caddy web server configuration for the home server (Mac Mini). Serves a dashboard landing page, reverse-proxies local services (Transmission), and holds optional links on the dashboard, such as your NAS admin pages. It has split internal/external access control.
Key Commands¶
# Validate Caddyfile (requires custom caddy build with cloudflare module)
# Use a dummy token — must look like a real token to pass the cloudflare module's format check
HOSTNAME=$(hostname -s) CF_API_TOKEN=dummy0token0for0validation0only000000000 caddy validate --config Caddyfile
# Deploy everything (web assets, Caddyfile, wrapper, plist) and validate
sudo ./caddy-setup.sh
# Restart Caddy (via LaunchDaemon — injects CF_API_TOKEN from System keychain)
sudo launchctl kickstart -k system/com.caddyserver.caddy
# Stop Caddy
sudo launchctl bootout system/com.caddyserver.caddy
# Health check all endpoints
caddy-health.sh
# Generate a bcrypt password hash for basic_auth
caddy hash-password
Important: Do not use caddy reload or caddy start directly — the Caddyfile
requires CF_API_TOKEN which is injected by caddy-wrapper.sh (run via the LaunchDaemon).
Always restart through launchctl kickstart.
Architecture¶
Caddyfile Structure¶
The Caddyfile uses four site blocks and three snippets:
- Catch-all
:443— Rejects direct IP access or unrecognized hostnames (403) - Internal block (
{$HOSTNAME}.local localhost <server LAN IP>(__SERVER_LAN_IP__, currently 192.168.21.10)) — Usestls internal(Caddy's built-in CA). Importscommon_config - External block (
example.com) — Uses Let's Encrypt DNS-01 via Cloudflare for public TLS. Importscommon_config. Kept as the rollback path after the move to the Cloudflare Tunnel - Tunnel block (
http://:8080,bind 127.0.0.1) — The Cloudflare Tunnel origin. Importssite_routesbehind an email-header check, andaccess_log. See below andcloudflare-tunnel.md
The (common_config) snippet holds the LAN/WAN access control, and imports (site_routes) and (access_log):
- Network matchers:
@local_networkand@external_networkbased on192.168.21.0/24 - Basic auth: Applied to
@external_auth_required: any non-LAN source IP, on both sites. It must never key onhost: the WAN forward reaches the internal site too, and the client chooses the Host/SNI - Media file browser:
/media/*reverse-proxied tolocalhost:9880(Python file server, local network only)
The (site_routes) snippet holds the routes every site serves:
- Reverse proxies:
/transmission/*to127.0.0.1:9091(Transmission publishes 9091 on loopback only) - Static files: Served from
/usr/local/var/www - Dashboard link redirects: optional links on the dashboard, such as your NAS admin pages. See Dashboard links
The (access_log) snippet holds the JSON access log (/usr/local/var/log/caddy/access.log). It is separate from site_routes because log is site-level only, and the tunnel site imports site_routes inside a handle.
Tunnel site (http://:8080)¶
cloudflared forwards example.com requests that carry a valid Cloudflare
Access token to http://127.0.0.1:8080. Every request on this listener comes
from loopback, so:
- It must not import
common_config. Its auth matcher exempts loopback, so it would apply no auth, and its/mediaguard (@not_local) would let every tunnel request through. /mediamust not be insite_routes, for the same reason. Without the handler,/mediaon the tunnel site falls through tofile_server(404).- Every route sits inside
handle @access_user, which matches only whenCf-Access-Authenticated-User-Emailis exactlyACCESS_ALLOWED_EMAIL(substituted bycaddy-setup.sh, which refuses an empty or malformed value). A catch-allhandlereturns 403. Nothing else may sit at site level: Caddy sortsredirandreverse_proxyahead ofhandle, so they would run before the check. - It uses no
routeblock:routeruns directives in written order, andfile_server(first insite_routes) would then answer every path. - Cloudflare Access, the cloudflared JWT check, the loopback bind and the policy-drift watchdog are the other layers.
- Optional: with
HOME_ASSISTANT_HOSTNAMEandHOME_ASSISTANT_UPSTREAMset, a second hostname on this site goes to Home Assistant, inside the same email check. That host serves only HA, and the main host never reaches it. With the keys empty, the render does not change. See Home Assistant.
tests/caddyfile.bats pins all of this. caddy validate opens the access
log, so run it as root, or on a copy with the log path pointed somewhere
writable.
Dashboard links¶
The dashboard can show up to two optional links, such as your NAS admin
pages. Each link has three keys in config.conf:
HOME_LINKn_NAME: the label on the dashboard.HOME_LINKn_PATH: the path on this server./<path>and/<path>/redirect (302) to the URL. Letters, digits and-only.HOME_LINKn_URL: the target. It must start withhttps://.
n is 1 or 2. HOME_LINKS_TITLE sets the heading (default Links). A link
with an empty URL is left out of the Caddyfile, the dashboard and
caddy-health.sh. With no links, the dashboard has no links section.
The redirects are in (site_routes), so on the tunnel site they sit behind
the allowed-email check like every other route.
caddy-setup.sh checks the keys before it writes anything, and stops on a
label, path or URL that could break the Caddyfile or the page. It also stops
on a path that is already a route (transmission, media) or that both
links use.
The old QUICKCONNECT_* keys are deprecated. If one is set and its link has
no HOME_LINKn_NAME or HOME_LINKn_PATH, caddy-setup.sh stops and prints
the lines to add. This keeps a deploy from changing the dashboard labels or
link paths without notice. When the new keys are set too, it uses them and
prints a deprecation warning. Remove the old keys after you move the values.
Environment Variables¶
HOSTNAME— Required. Used in site address ({$HOSTNAME}.local). Set viahostname -sCF_API_TOKEN— Required for Cloudflare DNS-01 cert issuance/renewal. Injected at runtime bycaddy-wrapper.shfrom the System keychain; never set in the plist or Caddyfile directlySYNOLOGY_HOST— Optional. Defaults topecorino.local
File Layout (in mac-server-setup)¶
app-setup/caddy-setup.sh— Full deployment script: copies web assets, Caddyfile, wrapper script, and LaunchDaemon plists to their deployed locations, then validates configapp-setup/templates/Caddyfile— Production config (the main file you'll edit)app-setup/templates/caddy-wrapper.sh— ReadsCF_API_TOKENfrom System keychain, exports it, thenexecs Caddy. Deployed to/usr/local/bin/caddy-wrapper.shapp-setup/templates/caddy-health.sh— Tests landing page, proxies, certs, and process status. Deployed to/usr/local/bin/caddy-health.shapp-setup/templates/media-server.py— Python HTTP file server for NFS media volume (binds to127.0.0.1:9880)app-setup/templates/www/— Static web root (dashboardindex.html, favicons, manifest)app-setup/templates/com.caddyserver.caddy.plist— Caddy LaunchDaemonapp-setup/templates/com.media-server.plist— Media server LaunchDaemonapp-setup/templates/caddy-root-ca.crt— Internal CA root certificate for client trust
Deployment Path¶
Files are not served directly from the repo. caddy-setup.sh deploys everything to the correct locations:
www/*→/usr/local/var/www/(web root)Caddyfile→/Users/operator/.config/caddy/Caddyfile(runtime config)caddy-wrapper.sh→/usr/local/bin/caddy-wrapper.sh(token injection wrapper)caddy-health.sh→/usr/local/bin/caddy-health.sh(health check script)media-server.py→/usr/local/bin/media-server.py(NFS media file server)LaunchDaemons/*.plist→/Library/LaunchDaemons/(system services)
After deploying, restart both services:
sudo launchctl kickstart -k system/com.caddyserver.caddy
sudo launchctl kickstart -k system/com.<hostname>.media-server
Media File Server¶
The /media/ path serves a file browser for the NAS NFS mount at /Users/operator/.local/mnt/<share>. Due to a macOS security restriction, Caddy cannot access NFS mounts directly — macOS blocks all launchd-spawned processes from NFS mounts unless the binary has Full Disk Access (FDA) granted in System Preferences → Privacy & Security → Full Disk Access.
Architecture: media-server.py (Python http.server) runs as a LaunchDaemon (com.<hostname>.media-server) on 127.0.0.1:9880. Caddy reverse-proxies /media/* to it. The @not_local matcher blocks external access with a 403.
Prerequisite: /usr/bin/python3 must have Full Disk Access granted (done 2026-03-25). Without FDA, the media server will start but return "No permission to list directory" errors.
# Check media server status
sudo launchctl print system/com.<hostname>.media-server | grep state
# Test directly
curl http://127.0.0.1:9880/
# Logs
cat /Users/operator/.local/state/caddy/media-server.log
Custom Caddy Build¶
The deployed Caddy binary includes the caddy-dns/cloudflare DNS challenge module, which is not present in the Homebrew stock binary.
Do not use xcaddy. xcaddy is not available as a Homebrew formula. Use Caddy's official pre-built download API instead:
# Dev machine — download arm64 binary with cloudflare module
curl -L "https://caddyserver.com/api/download?os=darwin&arch=arm64&p=github.com%2Fcaddy-dns%2Fcloudflare" \
-o /tmp/caddy-cloudflare
chmod +x /tmp/caddy-cloudflare
# Verify the module is present before deploying
/tmp/caddy-cloudflare list-modules | grep cloudflare
# Expected: dns.providers.cloudflare
rsync /tmp/caddy-cloudflare operator@your-server.local:/tmp/caddy-cloudflare
On YOUR_SERVER (as operator):
sudo cp /opt/homebrew/bin/caddy /opt/homebrew/bin/caddy.homebrew-backup
sudo cp /tmp/caddy-cloudflare /opt/homebrew/bin/caddy
sudo chmod +x /opt/homebrew/bin/caddy
/opt/homebrew/bin/caddy version
/opt/homebrew/bin/caddy list-modules | grep cloudflare
Caddy is pinned (brew pin caddy) to prevent brew upgrade from overwriting the custom build. Run brew pin caddy as the admin account (not operator) since Homebrew requires the admin account.
To update after a Caddy version bump: download a fresh binary from the API above, replace, and re-pin.
Check current binary: /opt/homebrew/bin/caddy list-modules | grep cloudflare (expect dns.providers.cloudflare)
TLS Strategy¶
LAN (Internal)¶
Caddy's internal PKI with 90-day intermediate certs (intermediate_lifetime 90d in global config). Clients must install caddy-root-ca.crt to trust these certs. The internal block matches {$HOSTNAME}.local localhost <server LAN IP> (__SERVER_LAN_IP__, currently 192.168.21.10) and uses tls internal.
WAN (Let's Encrypt DNS-01 via Cloudflare)¶
No inbound ports needed — the DNS-01 challenge proves domain ownership by creating a TXT record at _acme-challenge.example.com via the Cloudflare API rather than serving a file over HTTP.
Token injection chain:
System keychain (root-accessible)
→ caddy-wrapper.sh (reads via `security find-generic-password`)
→ CF_API_TOKEN env var (exported before exec)
→ Caddy process (reads via {env.CF_API_TOKEN} in Caddyfile)
→ caddy-dns/cloudflare module (makes Cloudflare API calls)
The token never appears in the LaunchDaemon plist (world-readable at /Library/LaunchDaemons/) or in the Caddyfile (readable by operator). The plist calls /bin/bash /usr/local/bin/caddy-wrapper.sh; the wrapper reads the token from the System keychain and execs Caddy.
Verify the LaunchDaemon is using the wrapper:
sudo launchctl print system/com.caddyserver.caddy | grep -A3 "program ="
# Expected:
# program = /bin/bash
# arguments = {
# /bin/bash
# /usr/local/bin/caddy-wrapper.sh
Verify the token is valid:
CF_TOKEN=$(sudo security find-generic-password \
-s cloudflare-api-token -a example.com -w \
/Library/Keychains/System.keychain)
curl -s -H "Authorization: Bearer $CF_TOKEN" \
"https://api.cloudflare.com/client/v4/user/tokens/verify" \
| python3 -m json.tool
unset CF_TOKEN
# Expected: "status": "active"
certmagic DNS Propagation: Two Knobs¶
The caddy-dns/cloudflare module uses certmagic internally. certmagic has two separate knobs for DNS propagation that are easy to confuse:
Environment Variable Syntax: {$VAR} vs {env.VAR}¶
Critical: There are two env var syntaxes in Caddy and they are NOT interchangeable in all contexts.
| Syntax | Processed by | When | Result in JSON config |
|---|---|---|---|
{$CF_API_TOKEN} |
Caddyfile adapter | At parse time | Actual token value baked in |
{env.CF_API_TOKEN} |
Caddy runtime | At request time (HTTP handlers) | Literal string passed to module |
The dns cloudflare directive is a TLS module config value, not an HTTP handler. The cloudflare DNS module receives whatever string is in api_token and uses it as a Bearer token. If {env.CF_API_TOKEN} is used, the literal text {env.CF_API_TOKEN} is passed to the module, Cloudflare rejects it as an invalid token, and the DNS challenge fails silently — no TXT record is ever created, and Let's Encrypt returns HTTP 403.
Always use {$CF_API_TOKEN} (dollar-sign form) for TLS/DNS module configuration.
You can verify the active config was substituted correctly:
curl -s http://localhost:2019/config/apps/tls | python3 -c "
import sys, json
cfg = json.load(sys.stdin)
policies = cfg.get('automation', {}).get('policies', [])
for p in policies:
if 'example.com' in p.get('subjects', []):
print(json.dumps(p, indent=2))
"
If api_token shows {env.CF_API_TOKEN} (literal), the substitution failed. If it shows the actual token value, it's correct.
propagation_timeout¶
Controls how long certmagic will poll the authoritative nameservers itself before giving up and signaling LE. Default is a few minutes.
Setting on YOUR_SERVER: propagation_timeout -1 disables certmagic's own propagation poll entirely. certmagic creates the TXT record via the Cloudflare API, then immediately signals LE that the challenge is ready — without checking propagation itself. Let's Encrypt performs its own independent check from their servers.
This avoids issues where certmagic's direct TCP/53 queries to Cloudflare's authoritative nameservers may be blocked by network configuration or VPN routing.
propagation_delay¶
A fixed sleep inserted after creating the TXT record via the API and before signaling LE that the challenge is ready. Default is 0 (no delay).
Why this matters: With propagation_timeout -1, certmagic notifies LE the instant the Cloudflare API call returns successfully. Cloudflare's authoritative nameservers may not have propagated the new TXT record to all resolvers by the time LE's servers query them. This creates a race: if LE polls before the record propagates, it gets a 403 "No TXT record found."
propagation_delay 30s is required on YOUR_SERVER. Without it, certmagic notifies LE immediately after the Cloudflare API call returns; LE's resolvers check before Cloudflare has propagated the record to its authoritative nameservers, resulting in HTTP 403 "No TXT record found" on every attempt. The current Caddyfile includes this.
example.com {
tls {
issuer acme {
dir https://acme-v02.api.letsencrypt.org/directory
dns cloudflare {$CF_API_TOKEN}
propagation_timeout -1
propagation_delay 30s
}
}
import common_config
}
certmagic retries cert issuance up to 3 times with 60-second gaps. A single propagation race is usually self-correcting on the first retry.
ACME State Directory Structure¶
Caddy stores ACME data under ~/Library/Application Support/Caddy/ (as whatever user it runs as — root for the LaunchDaemon, so /var/root/Library/Application Support/Caddy/, but the plist sets HOME=/Users/operator so it ends up at /Users/operator/Library/Application Support/Caddy/).
~/Library/Application Support/Caddy/
certificates/
local/ # Internal PKI certs — do NOT delete
acme-v02.api.letsencrypt.org-directory/ # Production LE certs + keys
acme-staging-v02.api.letsencrypt.org-directory/ # Staging LE certs + keys (if any)
acme/
acme-v02.api.letsencrypt.org-directory/ # Production ACME account keypair + registration
acme-staging-v02.api.letsencrypt.org-directory/ # Staging ACME account keypair (if any)
pki/
authorities/
local/ # Internal CA root + intermediate — do NOT delete
Critical: certificates/ and acme/ are separate trees. Deleting only certificates/ removes the issued cert but leaves the ACME account data in acme/. On the next startup, Caddy finds the existing account and reuses it — including any stale staging account. You must clear both directories for each CA you want to reset.
Clearing All ACME State (Both Staging and Production)¶
Run as operator (Caddy runs as root but writes to operator's home via the plist's HOME env var):
rm -rf "/Users/operator/Library/Application Support/Caddy/certificates/acme-staging-v02.api.letsencrypt.org-directory"
rm -rf "/Users/operator/Library/Application Support/Caddy/certificates/acme-v02.api.letsencrypt.org-directory"
rm -rf "/Users/operator/Library/Application Support/Caddy/acme/acme-staging-v02.api.letsencrypt.org-directory"
rm -rf "/Users/operator/Library/Application Support/Caddy/acme/acme-v02.api.letsencrypt.org-directory"
Verify only local/ remains:
ls "/Users/operator/Library/Application Support/Caddy/certificates/"
ls "/Users/operator/Library/Application Support/Caddy/acme/"
Staging Account Pitfall¶
If Caddy ever ran without an explicit dir in the Caddyfile (e.g., during early testing), certmagic may have registered an account against the staging CA. The staging account persists in acme/acme-staging-v02.... On subsequent startups, if dir is still not set, Caddy may find and reuse the staging account, issuing untrusted certs.
Always include an explicit dir pointing to the production CA:
Verifiable in the log: look for "ca":"https://acme-v02.api.letsencrypt.org/directory" (production) vs "ca":"https://acme-staging-v02.api.letsencrypt.org/directory" (staging).
Troubleshooting Cert Issuance¶
Symptom: staging CA used instead of production¶
Log shows acme-staging-v02.api.letsencrypt.org. Cause: stale staging ACME account in acme/ directory, and missing explicit dir in Caddyfile.
Fix:
- Add
dir https://acme-v02.api.letsencrypt.org/directoryto theissuer acmeblock - Clear both
acme/acme-staging*andacme/acme-v02*directories - Reload Caddy
Symptom: DNS propagation timeout (dial tcp <ip>:53: i/o timeout)¶
certmagic cannot reach Cloudflare's authoritative nameservers on TCP/53. This can be caused by network configuration, VPN routing, or firewall rules blocking direct DNS queries.
Fix: propagation_timeout -1 in the issuer acme block.
Symptom: "No TXT record found at _acme-challenge.example.com" (HTTP 403)¶
certmagic created the TXT record via the Cloudflare API but LE's resolvers checked before the record propagated from Cloudflare's authoritative NS.
Diagnosis:
- Verify the token is valid:
curl .../user/tokens/verify(see command above) - Verify the LaunchDaemon uses the wrapper:
sudo launchctl print system/com.caddyserver.caddy | grep -A3 "program =" - certmagic retries automatically (3 attempts, 60s apart) — a single failure is usually self-correcting
If it fails all 3 attempts: add propagation_delay 30s to the issuer acme block.
Reading cert issuance logs¶
tail -f ~/.local/state/caddy/caddy-error.log | grep -i 'example\.com\|acme\|certif\|cloudflare\|obtain\|error'
Key log messages in order of a successful issuance:
"msg":"obtaining certificate","identifier":"example.com"
"msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/..."
"msg":"trying to solve challenge","challenge_type":"dns-01"
"msg":"certificate obtained successfully","identifier":"example.com"