Appreciation post: That one server crash that taught me about backups

Last Tuesday our e-commerce site went down at 2 PM during a flash sale in Austin. Turns out our database backup script had been failing for 6 months without anyone checking. I spent 14 hours rebuilding customer orders from email receipts and PayPal logs.

2 comments

2 Comments

jana_shah1mo ago

6 months of failing" - IT guys make great horror movie villains.

jana_jones1mo ago

Wait, nobody checked the backup logs for 6 months? Oof. That's the kind of thing that only gets noticed when everything blows up. I had a similar close call with my small shop's database last year. Turns out the backup size was zero bytes for 4 months because I renamed a folder. Just never looked at the reports. Now I have a weekly calendar reminder to actually open the backup file and make sure it has data in it. Lesson learned hard way I guess.