The Uncomfortable Audit
I'm building Now I Get It! into a commercial product, so I sat down to audit what would happen if something went wrong. The answer was: everything would be gone.
Eight DynamoDB tables with no point-in-time recovery. No deletion protection -- an accidental delete-table or a botched stack deletion would wipe the data permanently. The S3 bucket holding every generated HTML page and blog post had no versioning -- overwrite a file and the old version vanishes. Lambda functions had no version history, so a bad deploy meant scrambling to redeploy from source with no quick rollback path.
For a side project, that's acceptable. For something heading toward paying customers, it's negligent.
What Changed
DynamoDB: Every table now has Point-in-Time Recovery enabled -- 35 days of continuous backups, recoverable to any second within that window. All eight tables also have deletion protection, which means you have to explicitly disable it before deleting a table. CloudFormation can't accidentally remove them during a stack update, and a stray CLI command won't destroy production data. The cost is roughly $0.20 per GB per month of backup storage. At current data volumes, we're talking under $2/month across all tables.
S3: The frontend bucket now has versioning enabled. Every previous version of every object is retained, so an overwritten HTML page or a deleted blog post is always recoverable. Enabling versioning is technically irreversible (you can suspend it, never fully disable it), but that's a feature, not a bug, when you're protecting user-generated content.
Lambda: Deploy scripts now publish an immutable version snapshot after every code update, tagged with a UTC timestamp. Old versions get pruned automatically, keeping the 10 most recent. This gives meaningful rollback depth without unbounded storage growth. The version publishing logic lives in the shared deploy library (deploy-common.sh), so it works identically across test and production deploys.
The Runbook
Enabling backups is only half the story. You also need to know how to use them when something actually breaks. I wrote a comprehensive runbook covering two DynamoDB recovery scenarios:
Scenario A is the simpler case -- a table gets deleted. You restore directly to the original table name (possible because the original no longer exists), then re-enable the settings that PITR doesn't automatically restore: PITR itself, deletion protection, TTL, and tags. No Lambda changes needed because the table name hasn't changed.
Scenario B is trickier -- data corruption where the table still exists. You restore to a temporary table, verify the data looks right, then swap Lambda environment variables to point at the restored table. The critical gotcha here is that the AWS CLI's --environment flag on update-function-configuration is a full replace, not a merge. Pass just the one variable you're changing and you'll wipe every other environment variable on the function. The runbook includes a table-to-Lambda reference map so you know exactly which functions need updating for each table.
The runbook also covers S3 version recovery, Lambda rollback procedures, full stack rebuild from scratch, and secret recreation.
The Rebase Surprise
This branch was created before a major refactor of the deploy scripts landed on main. My initial implementation added Lambda versioning directly to the old monolithic deploy scripts. Rebasing onto main produced conflicts because those scripts had been completely replaced with modular subcommands.
The resolution was actually cleaner than the original -- instead of duplicating versioning logic in two scripts, it went into a single function in the shared library. The rebase also pulled in two new DynamoDB tables (TakedownTable and EmailTemplatesTable) from work that had landed on main in the meantime, so they got PITR and deletion protection too. Went from 6 tables to 8 without having to go back and amend anything.
Lessons
- Backups you can't restore from aren't backups. Enabling PITR is one CloudFormation property. Knowing how to actually recover from it -- including which settings don't survive the restore, which Lambdas reference which tables, and how to swap traffic without wiping environment variables -- is a runbook.
- Infrastructure audits surface uncomfortable truths. The gap between "this works" and "this is recoverable" was enormous. None of these changes affected how the application runs day-to-day, but they completely change the blast radius of a mistake.
- Additive changes are the safest changes. Every one of these -- PITR, deletion protection, S3 versioning, Lambda version publishing -- is a purely additive property. Nothing was removed, replaced, or restructured. The risk was effectively zero, which is exactly what you want for changes that protect against risk.