Loading…
32
Our deploy broke checkout on a Friday afternoon and I still think about the 11 minutes
I run ops at a small SaaS company off South Congress here in Austin and back in April we pushed a pricing change at 4:40pm on a Friday, right before everyone logged off for the weekend. Checkout started throwing 500s within 11 minutes and our biggest customer (about $9k a month) emailed our CEO before I even saw the alert. We rolled back in 6 minutes but the damage was done, they wanted a call Monday morning and I spent the weekend writing up what happened. Does anyone here have a real rule for Friday deploys, or is it just don't do it and hope nobody breaks it?
1 comments
Log in to join the discussion
Log In1 Comment
the_anthony19d ago
An 11 minute outage that you fixed in 6 minutes and the customer still emails your CEO? That sounds less like a deploy problem and more like a customer who was already looking for a reason to be mad. Every shop I've worked at has had a bad Friday push at some point, and honestly most of them were way worse than 11 minutes of 500s on checkout. The real issue here isn't the timing, it's that a $9k a month account has a direct line to your CEO and used it over something that minor. A Friday deploy rule sounds great until you realize you can break prod just as easy on a Tuesday at 10am. What you actually need is better alerting and maybe a canary rollout, not some superstitious ban on shipping after lunch.
7