Backup failure alerts: know when your backups stop working

The dangerous backup is not the one that fails loudly. It is the one that has been running every night for eight months, producing a file, and writing nothing useful into it. Nobody noticed, because nothing ever complained. Proper backup failure alerts exist to remove that silence, and the second half of the job is alerting when a backup does not run at all.

This article covers the ways backups break without saying so, how to make a job report honestly, and why "no news" should never be treated as good news.

Why a broken backup stays invisible for months

Backups are unusual among systems in that nothing depends on them until the day everything does. A website that stops serving pages is noticed in minutes. A backup that stops being usable is noticed only when someone tries to restore it, which by definition is the worst possible moment to find out.

Two habits make it worse. The first is treating an empty inbox as confirmation, when it may just mean the job never reached the line that sends mail. The second is checking that a file exists rather than checking what is in it. A zero-byte archive and a good one both look like a file in a listing.

The quiet failure modes worth knowing

The disk fills up

Backups grow, and they are often written to the same volume as the site. When the disk reaches capacity, the archive is truncated mid-write. Depending on the script, the job may still finish and report success.

Credentials expire or rotate

A storage access key is rotated, a password policy forces a change, an application password for a mail or cloud account is revoked. The upload step fails with an authentication error that goes into a log nobody reads, while the local part of the job completes normally.

Paths change after a migration

The site moves to a new server or a new directory structure, and the backup script keeps pointing at the old path. It faithfully archives an empty or obsolete folder. New directories added later, such as a fresh uploads location, are simply never included because nobody updated the list.

The database dump goes wrong

A dump can fail partway through on a lock timeout, a permissions change on a new table, or a memory limit, leaving a file that starts correctly and stops halfway. Opened casually it looks like a valid dump, because the first few thousand lines are fine.

The schedule stops running

A server reboot, a control panel upgrade or an operating system update can drop a scheduled job. The task disappears rather than erroring, so there is nothing to alert on unless you are watching for absence.

The destination quietly rejects writes

A storage bucket's permissions change, a retention rule deletes older files, or the remote account runs out of space or credit. The job uploads, receives an error, and moves on.

The job outgrows its window

As data grows, a backup that took twenty minutes starts taking three hours and overlaps with the next run or with peak traffic. Overlapping runs can corrupt each other or get killed halfway.

Alert on failure, and also on absence

Most people configure the first half: if the script errors, send an email. That catches perhaps half of the real cases, because the most dangerous failures are the ones where the script never runs or never reaches its own error handling.

The fix is a heartbeat, sometimes called a dead man's switch. Your backup job sends a short signal to an external monitoring service only when it completes successfully. That service expects the signal within a defined window, and alerts you when it does not arrive. Silence becomes an alarm rather than a comfort.

Heartbeat monitoring has one important property: the watcher is outside the server. If the machine is down, out of disk, or has lost its scheduler entirely, the alert still fires, because it is triggered by the absence of a message rather than by anything the server has to do.

Send the heartbeat at the very end of the job, after the upload and after any verification step, never at the start. A signal sent on launch only tells you the job began.

Make the job report honestly

Scripts often hide their own failures. A few disciplines fix that:

• Check the exit status of every command, not just the last one. In a shell pipeline, a failure in the middle can be masked by a successful final command unless you configure it otherwise.
• Capture error output to the log instead of discarding it. A surprising number of backup scripts send errors to a device that ignores them.
• Have the job fail deliberately when a step fails, rather than continuing to the next stage with nothing to work on.
• Verify the archive can be opened before declaring success, and treat an unreadable archive as a failure.
• Write one summary line per run to a log: timestamp, duration, file size, destination and outcome. That single line answers most questions later.

Only after all of that should the job report success. The order matters: archive, upload, verify, then signal.

Where the alert should go

An alert that depends on the failing system is not an alert. If backup notifications are sent by the same server that is out of disk, they will not arrive.

Use an external mail service or a monitoring platform, deliver to a shared mailbox rather than one person, and add a second channel such as a team messaging group. Name a primary owner and a backup owner, and make sure both have access to the server and the storage account.

Keep the alert volume low enough that people still read it. Daily success emails train everyone to ignore the whole stream, and the one failure notice arrives in a folder nobody opens. Alert on failure and on absence; report success in a weekly digest if you want reassurance.

Write alerts a tired person can act on

At two in the morning, nobody wants to go hunting for basics. A useful alert names the site or server, the job, when the last successful run was, the error message or the missing heartbeat window, and where the logs are. Add the owner's name if more than one person could pick it up.

If your alert reads only "backup failed", improve the message before you improve anything else. It costs ten minutes and saves an hour every time it fires.

Break it on purpose, then fix it

An alerting setup you have never seen fire is a hypothesis. Test it deliberately on a staging server or during a quiet window: rename the credentials file and confirm the failure alert arrives, then disable the scheduled job for a cycle and confirm the absence alert arrives too. Both paths need to be proven, because they are handled by different mechanisms.

Once they work, note the date, and retest after any change to the server, the storage provider or the team.

Check your last successful backup today

Do one thing before closing this page: find your most recent backup file, check its timestamp and its size, and ask whether either surprises you. If you cannot find it, or nobody knows where it is written, that is the answer to how well the alerting works.

Backup jobs and their alerting are part of routine server care rather than a one-off project. Our team sets this up on the servers we run under managed Linux hosting, and can review an existing arrangement through the support desk if you would rather have someone else watching it.

Frequently asked questions

Is a daily success email enough to know my backups work?

No, for two reasons. It does not prove the archive contains anything useful, and it goes missing silently when the job never runs. Pair a failure alert with a heartbeat check so the absence of a message is treated as a problem.

Should backups be stored on the same server as the website?

Keep at least one copy off the server. A local copy is convenient for quick restores, but it disappears along with everything else if the disk fails, the account is suspended or the machine is compromised.

My host says backups are included. Do I still need alerts?

Ask what the service actually covers: how often it runs, how long copies are kept, whether you can download them, and whether you are told when one fails. Many included plans are best-effort. If you cannot get clear answers, run your own backup alongside it.

How quickly should I be told about a failed backup?

Within one backup cycle. If backups run nightly, a failure should reach someone the next morning, so at most one day of changes is at risk. Waiting until the weekly review means a week of work is unprotected.