Skip to content

Monitor database backups with a size check

A backup job can succeed and still leave you with nothing useful: an empty database dump after a credentials change, a truncated file after the disk filled, or yesterday’s file copied again. A heartbeat check with a numeric value catches these along with the usual failures: the job reports the backup’s size after it succeeds, and Pingstack alerts you when the report is late or the size drops below what you expect.

1. Create a heartbeat check with a size threshold

Section titled “1. Create a heartbeat check with a size threshold”

Create a heartbeat check whose schedule matches the backup job, with a grace period longer than the backup’s usual runtime.

Then turn on value alerting:

  • Reported value alerting: below a threshold
  • Warning value: somewhat below your usual backup size, in megabytes
  • Critical value: a size that would mean something is badly wrong

For below, the critical value must be less than the warning value, and a value equal to a threshold counts as a breach. If your backups are usually around 800 MB, a warning at 600 and a critical at 100 is a reasonable start. See heartbeat checks for how values and check-ins combine.

Attach a notification group.

Copy the location’s ping URL from the check in Pingstack and save it in a file only the backup user can read:

Terminal window
install -m 600 /dev/null ~/.pingstack-backup
read -rs URL && printf 'PINGSTACK_PING_URL=%s\n' "$URL" > ~/.pingstack-backup
/opt/jobs/pg-backup.sh
#!/usr/bin/env bash
set -euo pipefail
source ~/.pingstack-backup
FILE="/backups/app-$(date +%F).dump"
pg_dump --format=custom --file="$FILE" app_production
# Confirm the archive is readable before calling it a success
pg_restore --list "$FILE" > /dev/null
SIZE_MB=$(( $(stat -c %s "$FILE") / 1024 / 1024 ))
curl --fail --silent --show-error --max-time 10 --retry 3 \
--get --data-urlencode "value=$SIZE_MB" "$PINGSTACK_PING_URL" > /dev/null
/opt/jobs/mysql-backup.sh
#!/usr/bin/env bash
set -euo pipefail
source ~/.pingstack-backup
FILE="/backups/app-$(date +%F).sql.gz"
mysqldump --single-transaction --routines app_production | gzip > "$FILE"
# Confirm the compressed file is intact
gzip --test "$FILE"
SIZE_MB=$(( $(stat -c %s "$FILE") / 1024 / 1024 ))
curl --fail --silent --show-error --max-time 10 --retry 3 \
--get --data-urlencode "value=$SIZE_MB" "$PINGSTACK_PING_URL" > /dev/null

set -euo pipefail stops the script at the first failure, including a mysqldump failure on the left of the pipe, so a failed backup sends nothing. On macOS, use stat -f %z instead of stat -c %s.

Database credentials belong in ~/.pgpass or ~/.my.cnf with mode 600, not in the script.

Schedule the script with cron, a systemd timer or a Kubernetes CronJob.

Run the script by hand as the backup user. Open the check in Pingstack and confirm the location received a heartbeat with the size you expect, and shows a next deadline.

A backup on the same disk as the database doesn’t survive losing that disk. If you upload backups to object storage, report after the upload succeeds, so the check covers the whole job.

  • Thresholds set too tight. A database that shrinks after a cleanup can cross the warning value. Adjust the thresholds as the data changes.
  • Slow growth. A backup that grows over time eventually outlasts the grace period. Lengthen it.
  • Compression changes. Switching formats or compression levels changes the size. Update the thresholds when you do.

If alerts still don’t match what you expect, see troubleshooting monitoring.