We take regular system backups, and our Terms are explicit that they are a courtesy rather than a guarantee. That isn’t us dodging responsibility. Every provider is in the same position, whether or not they say so out loud. Provider-side backups protect against provider-side failures. They do a poor job of protecting against you.
The failures that actually destroy data are rarely hardware. They are a bad migration, a mistyped rm, a ransomware payload, or a bug that quietly corrupts records for three weeks before anyone notices.
The 3-2-1 rule
It is old advice because it keeps being correct:
- Three copies of your data: the live one and two backups.
- On two different types of storage or systems.
- With one copy off-site, meaning not on the same machine or under the same account as the original.
The off-site copy is the part people skip, and it is the part that matters when an account is compromised or a whole machine is lost. A backup stored on the server it is backing up isn’t a backup.
Snapshots aren’t backups
A snapshot is a point-in-time image of the disk, typically stored in the same infrastructure as the disk itself. It is excellent for what it is designed for: rolling back a risky upgrade five minutes after it went wrong. It isn’t durable, independent storage, and it usually shares a failure domain with the original.
A snapshot covers you for the change you made five minutes ago. It won’t cover you for the corruption that started three weeks ago.
Back up databases with the database, not the filesystem
Copying a database’s data directory while the database is running produces a file that may or may not be consistent, and you will find out which at the worst possible time. Use the tools the engine provides:
# PostgreSQL pg_dump -Fc mydb > /backups/mydb-$(date +%F).dump # MySQL / MariaDB mysqldump --single-transaction --quick mydb | gzip > /backups/mydb-$(date +%F).sql.gz
Automate it, then verify the automation
A backup job that has been silently failing for two months is worse than no backup job, because you believed you were covered. Whatever you use, whether that is restic, borg, or rsync to remote storage, make failure visible. Alert on the absence of a successful run, not just on errors, since a job that never starts produces no errors at all.
The restore drill
This is the step almost everyone skips, and it is the only one that proves any of the rest worked. Once a quarter, spin up a throwaway server and restore into it from scratch. You are testing for:
- Whether the archive is actually readable and complete.
- Whether you still have the encryption passphrase, and whether it is stored somewhere the disaster wouldn’t also destroy.
- How long a full restore actually takes, which is your real recovery time as opposed to the one you assume.
- Whether anything essential was never in scope: TLS certificates, cron jobs, environment files, firewall rules.
That last point catches people constantly. Application data gets backed up because it is obviously important. The configuration that makes the application run gets rebuilt from memory at 3am, badly.
Run the drill once and you will find at least one thing wrong. That is the entire point of running it while nothing is on fire.