Monitoring: Watchdogs and Alert Email¶
The server's watchdogs are small poll-and-exit scripts run by operator
LaunchAgents. When something goes wrong they email MONITORING_EMAIL through
msmtp (Gmail SMTP). They all send that email through one shared library,
alert-lib.sh.
Watchdogs¶
| Watchdog | LaunchAgent | Interval | Log | Deployed by |
|---|---|---|---|---|
plex-watchdog — Plex settings drift, Plex unreachable (restarts it after 3 polls), Plex cannot read media files, Plex data directory link |
com.<host>.plex-watchdog |
300 s, RunAtLoad | ~/.local/state/plex-watchdog.log |
plex-watchdog-setup.sh |
pia-port-watchdog.sh — PIA port forwarding lost |
com.<host>.pia-port-watchdog |
900 s | ~/.local/state/<host>-pia-port-watchdog.log |
podman-transmission-setup.sh |
stall-watchdog.sh — server blocked on a privacy prompt, a long transmission-done, supervisor failing |
com.<host>.stall-watchdog |
120 s, RunAtLoad | ~/.local/state/<host>-stall-watchdog.log |
podman-transmission-setup.sh |
access-policy-watchdog — Cloudflare Access rule for example.com changed, missing, or unreadable |
com.<host>.access-policy-watchdog |
900 s, RunAtLoad | ~/.local/state/access-policy-watchdog.log |
access-policy-watchdog-setup.sh |
tunnel-watchdog — Cloudflare Tunnel not ready (/ready not 200 for 3 checks) |
com.<host>.tunnel-watchdog |
60 s, RunAtLoad | ~/.local/state/tunnel-watchdog.log |
tunnel-watchdog-setup.sh |
Paths are under the operator's home (/Users/operator). All the LaunchAgents run
the script as /bin/bash <script> and send stdout and stderr to the same log
file. For the details of each check, see plex-watchdog.md,
pia-vpn.md, cloudflare-tunnel.md ("Drift watchdog"), and
stall-watchdog below.
Other files:
~/.local/lib/alert-lib.sh— the shared library (mode 0644)~/.config/msmtp/config— msmtp account, with an embedded Gmail App Password (mode 600). Do not print this file.~/.local/state/msmtp.log— msmtp's own log: one line per send attempt, withexitcode=EX_OKon success
The launchd PATH problem (fixed 2026-09)¶
launchd starts every agent with PATH=/usr/bin:/bin:/usr/sbin:/sbin. Homebrew
(/opt/homebrew/bin on Apple Silicon) is not on it. msmtp is a Homebrew binary.
Before this fix, both watchdogs sent mail with a bare
msmtp ... 2>/dev/null. Under launchd, msmtp was "command not found" on
every run, and 2>/dev/null discarded that error. So every alert from a
LaunchAgent failed silently, from the first watchdog deploy (plex-watchdog,
2026-03) until this fix. Nothing reached msmtp.log either, because msmtp
never started.
It stayed hidden because every manual check used a login shell.
msmtp-setup.sh sent its test email through sudo -iu operator msmtp ..., and
a login shell has Homebrew on PATH. So the setup test passed and the
LaunchAgents failed.
The fix has three parts:
- Each watchdog sets PATH itself, near the top:
export PATH="${HOMEBREW_PREFIX}/bin:/usr/bin:/bin:/usr/sbin:/sbin", withHOMEBREW_PREFIXderived from the CPU architecture. This is the same pattern astransmission-trigger-watcher.sh. alert_sendfinds msmtp by absolute path. When a send fails, it logs anERRORline that includes msmtp's stderr.msmtp-setup.shsends its test email throughalert_sendunderenv -i PATH=/usr/bin:/bin:/usr/sbin:/sbin, which is the launchd environment. If launchd would fail, setup fails too.
Also: the LaunchAgents use /bin/bash, which is bash 3.2. The Plex-unreachable
email used ${HOSTNAME_LABEL,,}, which bash 3.2 rejects as a "bad
substitution", so that email was never built. It now uses tr. Code that the
watchdogs run, including alert-lib.sh, must stay bash 3.2 compatible.
alert-lib.sh¶
app-setup/templates/alert-lib.sh is a library: source it, do not run it. It
sets no shell options and does not change PATH. It uses jq and date from
the caller's PATH.
The caller sets:
MONITORING_EMAIL— the recipientSTATE_FILE— the JSON state file, for the state and transition functions. Its directory must exist.log()— optional. If defined, the library writes its log lines through it (so they use the watchdog's log format). If not, they go to stderr.
Optional overrides:
ALERT_MSMTP— msmtp binary. Default<HOMEBREW_PREFIX>/bin/msmtp.HOMEBREW_PREFIX— default/opt/homebrewon arm64, else/usr/localALERT_MSMTP_CONFIG— default~/.config/msmtp/configALERT_REMINDER_SECONDS— used byalert_transition(see below)
Functions:
| Function | Behaviour |
|---|---|
alert_send SUBJECT BODY |
Sends one email (Subject: and To: headers, blank line, body). Returns 0 only if msmtp accepted it. On failure it logs an ERROR line with the reason (msmtp missing at <path>, config missing, or msmtp's exit code and stderr) and returns 1. |
alert_state_read |
Prints the state JSON, or {} if there is no state file. |
alert_state_get KEY [DEFAULT] |
Prints .KEY from the state, or DEFAULT if the value is missing, null or false. |
alert_state_write JSON |
Writes the state atomically (a temp file, then mv). |
alert_transition KEY IS_BAD SUBJECT BODY [RECOVERY_SUBJECT [RECOVERY_BODY]] |
Sends one alert when a condition goes bad, and one recovery email when it clears. See below. |
alert_transition keeps its state in .transitions[KEY] =
{alerted, since, last_sent} (since and last_sent are epoch seconds). It
does not change other keys in the state file, so you can call it for more than
one KEY in one run.
- Bad, not yet alerted: send the alert. Set
alertedonly if the send succeeded. If the send fails, it tries again on the next call. - Bad, already alerted: no email. But if
ALERT_REMINDER_SECONDSis set and that many seconds have passed sincelast_sent, it sends the alert again with[reminder]before the subject. - Good, was alerted: send the recovery email (best effort), then remove the
entry. The default subject is
RESOLVED: <SUBJECT>. - Good, not alerted: remove the entry, if one exists.
Each accepted alert or reminder logs ALERT sent: <KEY>: <subject>, and each
recovery logs RESOLVED: <KEY>, so the watchdog's own log shows every email
without cross-checking msmtp.log.
It returns 1 if an alert or reminder send failed, else 0. Under set -e, call
it as alert_transition ... || true.
transmission-done also calls alert_send, once per download that it cannot
identify or process. It is not a watchdog, and it reads the recipient from
alerts.email in its own config.yml. See "Email when a download cannot be
processed" in transmission-filebot.md.
plex-watchdog and pia-port-watchdog use alert_send and the state functions,
but keep their own alert, dedupe and heartbeat logic. The exceptions are
plex-watchdog's media access check (key media_unreachable) and data
directory link check (key data_link), which use alert_transition. stall-watchdog,
access-policy-watchdog and tunnel-watchdog use alert_transition for all of their checks.
Send a test alert under launchd's PATH¶
Run this as an admin on the server. It uses the same environment that launchd gives an agent:
sudo -u operator /usr/bin/env -i \
PATH=/usr/bin:/bin:/usr/sbin:/sbin \
HOME=/Users/operator \
MONITORING_EMAIL=you@example.com \
/bin/bash -c 'cd ~ && . ~/.local/lib/alert-lib.sh && alert_send "[YOUR_SERVER] test" "launchd-PATH test"'
Replace you@example.com with the MONITORING_EMAIL value from
app-setup/config/config.conf. On success the command prints nothing and exits
0. On failure it prints an [alert-lib] ERROR: ... line to stderr.
Then read the last line of msmtp's log. It must show exitcode=EX_OK:
Use sudo -u, not sudo -iu. A login shell adds Homebrew to PATH, and the
test then proves nothing about launchd.
stall-watchdog¶
Issue #199. On 2026-09-17 a macOS privacy (TCC) prompt about network-volume
access stayed open on the desktop for 19 hours. FileBot, the Transmission
container and Plex waited on it, and nothing alerted, because no check looked
for "blocked" as opposed to "failed". stall-watchdog runs every 2 minutes and
makes one alert_transition call per check:
| Key | Bad when | Notes |
|---|---|---|
tcc_prompt |
a TCC prompt has been open for 5 minutes or more | The email lists every open prompt: service, binary, how long it has been open. Reminder every 12 hours. |
tcc_log_unreadable |
log show failed 3 runs in a row |
State is left untouched on a failed read, so no prompt is lost. |
done_running_long |
a transmission-done process has run for over 45 minutes |
Found with ps -axo pid=,etime=,command=. Never killed: stopping FileBot mid-move can lose the file. |
supervisor_failing |
the status file shows 3 or more consecutive failed cycles | See "Supervisor status file" below. |
supervisor_stale |
the status file has not been updated for an hour | The supervisor loop is hung, or its agent is not running. |
ALERT_REMINDER_SECONDS is 43200 for all of these. State is in
~/.config/stall-watchdog/state.json.
How prompts are detected¶
The per-user tccd logs two lines per prompt, with the same msgID:
AUTHREQ_PROMPTING: msgID=17928.48, service=kTCCServiceSystemPolicyNetworkVolumes, subject=Sub:{/usr/local/stable/bash/bin/bash}Resp:{...}
AUTHREQ_RESULT: msgID=17928.48, authValue=2, ...
The first is logged when the dialog opens, the second when someone answers it
(Allow or Don't Allow). Each run reads new AUTHREQ_PROMPTING lines since the
previous run (with 1 minute of overlap, and at most 1 hour of catch-up after a
gap), and records each prompt in .tcc_prompts. While a prompt is open, it
also reads AUTHREQ_RESULT lines over the same window. On YOUR_SERVER there are
about 39,000 RESULT lines a day, mostly from the system tccd, and about one
PROMPTING line, so RESULT lines are read only when needed.
Prompts are keyed <tccd pid>/<msgID>. The msgID is <client pid>.<sequence>,
and the same client (sandboxd) talks to both the system tccd and the
per-user one, each with its own sequence. A RESULT from the system tccd can
carry the same msgID as an open prompt, and must not close it.
A prompt closes on its own RESULT line, or when the tccd process that owns it
has exited. It is never dropped for being old: that would send a recovery email
while the dialog is still on screen. tccd log lines do not stay in the log
for long (the 09-17 lines were already gone by 09-23), which is why the
watchdog keeps its own state.
Clearing a prompt by hand¶
If a RESULT line is missed (the log was unreadable for longer than the 1-hour
catch-up window), the prompt stays open and a reminder goes out every 12 hours.
After you check that no dialog is on the desktop, remove the entry. The recovery
email goes out on the next run. The path is spelled out because sudo -u
keeps your own HOME, so ~ would point at the wrong file:
sudo -u operator /bin/bash -c '
f=/Users/operator/.config/stall-watchdog/state.json
jq ".tcc_prompts" "$f" # find the key
jq "del(.tcc_prompts[\"<key>\"])" "$f" >"$f.tmp" && mv "$f.tmp" "$f"'
Supervisor status file¶
The podman supervisor loop (the podman-machine-start.sh wrapper, written by
podman-transmission-setup.sh) rewrites
~/.local/state/<host>-supervisor-status.json once per cycle:
A cycle fails when ensure_machine or ensure_container fails, or when a VM
recovery restart fails. A data-access failure on its own does not count: that
check already escalates by cycling the VM. The wrapper sends no email.
A missing file means "unknown", and nothing is alerted. The wrapper change
takes effect only when the supervisor is restarted, and restarting it restarts
the VM. The header of the wrapper heredoc in podman-transmission-setup.sh
explains why the loop must never exit. So stall-watchdog can be deployed first, and the supervisor checks
start working at the next VM maintenance window.
Deploying¶
Re-running all of podman-transmission-setup.sh also manages the container.
To deploy only the watchdog: render the template with the real
__SERVER_NAME__ and __MONITORING_EMAIL__, install it to
~operator/.local/bin/stall-watchdog.sh (mode 755), write the plist from
section 9e of the setup script, and load it:
sudo launchctl bootstrap gui/$(id -u operator) \
/Users/operator/Library/LaunchAgents/com.<host>.stall-watchdog.plist
To reload after changing the plist, bootout the label first. To rerun it
now, sudo launchctl kickstart -kp gui/$(id -u operator)/com.<host>.stall-watchdog.
Live test (2026-09-23)¶
Run on YOUR_SERVER with an ad-hoc-signed copy of Homebrew bash at
/Users/Shared/tcc-canary/probe2, started by a temporary operator
LaunchAgent that ran /bin/ls on the NAS mount:
| Time | Event |
|---|---|
| 15:15:03 | tccd logged AUTHREQ_PROMPTING; the dialog opened and ls blocked |
| 15:16:20 | supervisor: /data/ inaccessible inside container (failure 1/3) |
| 15:16:46 | the watchdog recorded the prompt (446/17928.161) |
| 15:18:42 | OpenVPN was signalled and the container stopped |
| 15:20:56 | alert email sent (first run after 5 minutes open) |
| 15:21:20–15:24:36 | supervisor: podman run -d hung for 180 s and was killed |
| 15:26:47 | Don't Allow clicked; ls failed with "Operation not permitted"; the container started within the same second |
| 15:27:04 | the watchdog closed the prompt ("open 11m") and logged RESOLVED: tcc_prompt |
| 15:27:06 | recovery email sent |
An earlier canary, answered with Allow after 34 seconds, was recorded and closed without an email, as intended.
An open prompt blocks every later network-volume check, not just the process
it was raised for. Requests reach the per-user tccd through sandboxd
(pid 17928 here, the first half of every msgID), which sends them one at a
time. Right after the canary's RESULT (17928.161), tccd logged the next
request, 17928.166: vfkit, with our stably-signed gtimeout as the
responsible process. It was allowed (authValue=2) the moment it was
evaluated, but it had waited behind the prompt, and with it the VM's access to
the NAS. Transmission was down from about 15:18 until the click. This is
consistent with the 09-17 outage, where FileBot, Transmission and Plex all
stalled behind one prompt.
A spot check 28 seconds into the first canary's prompt (15:12:44) found
podman ps, the VM's /data, Plex checkFiles=1 and operator's ls of the
NAS all answering in under a second. That check did not hit a fresh TCC
evaluation, so it proves nothing about blocking. Why some accesses need a
fresh evaluation and others do not is not known.
Expect an outage of Transmission, and possibly of Plex, for as long as a test prompt stays open.
If a prompt blocks Plex's own reads, plex-watchdog's media access check
([<host>] Plex cannot read media files, see plex-watchdog.md) is
expected to alert after 10 minutes. This is not tested against a real
prompt, and the spot check above shows checkFiles=1 can still answer while
a prompt is open.
To repeat the test, use a new path and signing identifier each time: once a
prompt is answered, tccd stores the answer for that binary and does not ask
again. tccutil reset takes a bundle ID and cannot remove these rows, but
they are inert once the binary is deleted.
Not yet proven¶
- Whether a prompt is logged again after a long gap. A scan looks back at most an hour. If the agent was not running for longer than that while a prompt opened, the watchdog sees that prompt only if tccd logs it again, which is expected on the next access attempt but not verified.
Troubleshooting¶
No alert email arrived. Look for ERROR lines in the watchdog's log. The
line tells you the cause:
msmtp not found or not executable at <path>— msmtp is not installed at that path. Runbrew install msmtp, or re-runmsmtp-setup.sh.msmtp config not found— runmsmtp-setup.sh.msmtp (<path>) exited <n> sending '<subject>': <stderr>— msmtp ran and failed. The stderr text gives the reason, for example an authentication failure after the App Password was revoked. Also seemsmtp.log.alert library not found at ~/.local/lib/alert-lib.sh— the watchdog stops (exit 1) on every run. Re-run the setup script that deploys it (msmtp-setup.sh,plex-watchdog-setup.shorpodman-transmission-setup.sh).
Where a failed alert goes. pia-port-watchdog does not set alerted when
its send fails, so it tries again every cycle while the outage continues.
plex-watchdog keeps its original behaviour: it records the drift even if the
send failed, so a failed drift email is not sent again. The ERROR line in the
log is the record of the failure.
Tests¶
bats tests/alert-lib.bats
bats tests/plex-watchdog.bats
bats tests/plex-data-link.bats
bats tests/pia-port-watchdog.bats
bats tests/stall-watchdog.bats
bats tests/msmtp-setup.bats
bats tests/podman-machine-start.bats # supervisor status file
bats tests/access-policy-watchdog.bats
bats tests/tunnel-watchdog.bats
Each watchdog test file has a "launchd PATH" test. It runs the rendered
watchdog as env -i PATH=/usr/bin:/bin:/usr/sbin:/sbin /bin/bash <script>,
with the msmtp mock only in a fake Homebrew prefix, and checks that the alert
is sent. These tests fail against the pre-fix templates.