Monitor a Kubernetes CronJob

A CronJob can fail in ways nobody sees: the job’s pods crash and exhaust their retries, the controller skips runs after a long outage, an image pull fails, or someone suspends the CronJob and forgets. A heartbeat check catches all of these: the job reports to Pingstack after it succeeds, and Pingstack alerts you when that report doesn’t arrive on time.

By the end of this recipe you’ll have a heartbeat check that expects the CronJob’s schedule, a Kubernetes Secret holding its ping URL, and a CronJob that reports each successful run.

1. Create a heartbeat check that matches the schedule

Find the schedule in your CronJob:

spec:
  schedule: "0 2 * * *"

Create a heartbeat check and set its schedule to Cron with the same expression. Without spec.timeZone, the schedule follows the timezone of the cluster’s controller manager, which is usually UTC. Set timeZone explicitly and choose the same timezone in Pingstack. See schedules and timezones.

Set a grace period that covers pod scheduling, image pulls, the job’s runtime and any retries you allow with backoffLimit. Attach a notification group.

2. Store the ping URL in a Secret

Copy the location’s ping URL from the check in Pingstack. Create the Secret without putting the URL in your shell history:

read -rs URL
kubectl create secret generic pingstack-nightly-report \
  --from-literal=ping-url="$URL"
unset URL

Keep the URL out of manifests you commit. If you manage Secrets with a tool such as Sealed Secrets or External Secrets, store it there instead.

3. Report success after the job’s work

The simplest pattern runs your work in an init container and the report in the main container. Kubernetes starts the main container only after every init container exits successfully, so a failed run never reports. Your job’s image doesn’t need curl either.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 2 * * *"
  timeZone: "Etc/UTC"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 600
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          initContainers:
            - name: report
              image: registry.example.com/reports:1.4.0
              command: ["./run-report"]
          containers:
            - name: pingstack
              image: curlimages/curl:8.10.1
              command: ["sh", "-c"]
              args:
                - curl --fail --silent --show-error --max-time 10 --retry 3 "$PINGSTACK_PING_URL" > /dev/null
              env:
                - name: PINGSTACK_PING_URL
                  valueFrom:
                    secretKeyRef:
                      name: pingstack-nightly-report
                      key: ping-url

If your image already has a shell and curl, you can instead chain the report onto the command in a single container:

command: ["sh", "-c", "./run-report && curl --fail --silent --show-error --max-time 10 \"$PINGSTACK_PING_URL\""]

concurrencyPolicy: Forbid stops a slow run from overlapping the next one. startingDeadlineSeconds lets the controller still start a run that’s a few minutes late, for example after a control-plane restart.

4. Confirm the first report

Start a run now instead of waiting for the schedule:

kubectl create job --from=cronjob/nightly-report nightly-report-test
kubectl logs -f job/nightly-report-test --all-containers

Open the check in Pingstack and confirm the location received a heartbeat and shows a next deadline. Delete the test job when you’re done.

Optional: report a number

Send a count the job produces as a numeric value and set warning and critical thresholds on the check. With the single-container pattern, capture the count from your command and add --get --data-urlencode "value=$COUNT" to the curl call.

Several CronJobs or clusters

Give each CronJob its own check and Secret. If the same CronJob runs in several clusters and each run matters, add a named location per cluster, so one healthy cluster can’t hide another’s silence.

Why a healthy CronJob might still alert

• Timezone mismatch. The check and spec.timeZone must agree.
• Pending pods. A job waiting for capacity or a pulled image can run past the grace period. Lengthen it, or fix the scheduling delay.
• Retries. With backoffLimit, a run that succeeds on its third attempt reports late. Allow for retries in the grace period.
• Suspended CronJob. spec.suspend: true stops all runs. Pingstack will alert, which is usually what you want.
• Missed schedules. If the controller misses too many scheduled times, for example after a long outage without startingDeadlineSeconds, it stops starting new runs until you intervene.

If alerts still don’t match what you expect, see troubleshooting monitoring.