Neil Craig on Nostr: Hands up if you caused a global outage today... Just me? Sorry! I was making a change ...

Hands up if you caused a global outage today...
Just me?
Sorry!

I was making a change to our "outside the UK" CDN config today for www.bbc.co.uk & www.bbc.com & the change included 2 bugs which pre-testing didn't spot:
- A regex typo which caused 404s on www.bbc.co.uk
- An incorrect TLS cert on the CDN origin which caused 503s on www.bbc.com

These caused ~7 minutes of significant global outage.

I spent most of the afternoon writing tests to catch this for next time.

#DevOps #WebDev #CDN