Monitor a systemd timer

systemd timers are reliable, but nobody reads the journal. A unit fails, the timer was never enabled on the rebuilt server, or the machine was off when the job should have run. A heartbeat check catches all of these: the service reports to Pingstack after it succeeds, and Pingstack alerts you when that report doesn’t arrive on time.

1. Create a heartbeat check that matches the timer

Find the timer’s schedule:

/etc/systemd/system/backup.timer

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

Create a heartbeat check and set its schedule to Cron with the equivalent expression (0 2 * * * for *-*-* 02:00:00). Run systemd-analyze calendar '*-*-* 02:00:00' to see the next run time.

OnCalendar uses the server’s local timezone unless you add one, such as *-*-* 02:00:00 UTC. Choose the same timezone in Pingstack. See schedules and timezones.

Set a grace period longer than the job’s usual runtime. If the timer uses RandomizedDelaySec, add that too. Attach a notification group.

For timers that use OnUnitActiveSec or OnBootSec instead of a calendar, use an Interval schedule on the check.

2. Keep the ping URL in a private environment file

Copy the location’s ping URL from the check in Pingstack and save it where only root can read it:

sudo install -d -m 700 /etc/pingstack
sudo install -m 600 /dev/null /etc/pingstack/backup.env
read -rs URL && printf 'PINGSTACK_PING_URL=%s\n' "$URL" | sudo tee /etc/pingstack/backup.env > /dev/null

read -rs waits for you to paste the URL without echoing it, then press Enter.

3. Report success after the service runs

Add the environment file and an ExecStartPost line to the service:

/etc/systemd/system/backup.service

[Unit]
Description=Nightly backup

[Service]
Type=oneshot
EnvironmentFile=/etc/pingstack/backup.env
ExecStart=/opt/jobs/backup.sh
ExecStartPost=/usr/bin/curl --fail --silent --show-error --max-time 10 --retry 3 ${PINGSTACK_PING_URL}

With Type=oneshot, systemd runs ExecStartPost only after ExecStart exits successfully, so a failed run sends nothing and Pingstack alerts you when the deadline passes.

Reload systemd and make sure the timer is enabled:

sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer

Or use the pingstack command

With the pingstack command installed, wrap ExecStart instead of adding ExecStartPost. It also reports failures right away with the end of the job’s output, plus each run’s time:

ExecStart=/usr/local/bin/pingstack run -- /opt/jobs/backup.sh

4. Confirm the first report

Start the service by hand:

sudo systemctl start backup.service
systemctl status backup.service

Open the check in Pingstack and confirm the location received a heartbeat and shows a next deadline. systemctl list-timers shows when the timer runs next.

Optional: report a number

ExecStartPost can’t see the job’s output, so report from the end of the script instead and remove the ExecStartPost line. See the database backup recipe for a script that sends the backup size as a value.

Why a healthy timer might still alert

• Machine off or asleep. With Persistent=true, a missed run starts at the next boot and reports then, after Pingstack has already alerted. That alert tells you the run was missed.
• Timezone mismatch. The check and OnCalendar must agree.
• Timer not enabled. A unit file that exists but isn’t enabled never runs. Check systemctl list-timers.
• Randomized delay. RandomizedDelaySec can push a run past a short grace period.

If alerts still don’t match what you expect, see troubleshooting monitoring.