Oliver Leaver-Smith - On how "just a monitoring change" took down the entire site and resilience engineering - #5

Bio

Oliver Leaver-Smith, better known as Ols, is a distinguished technologist currently working as a Senior Devops Engineer at Sky Betting and Gaming. His topics of expertise include OpenBSD, automation, chaos and resilience engineering, and Nerf warfare. He is also interested in security, privacy, opensource, decentralisation and federation, hardware hacking, and cyberdecks. Interested in technology (specifically how it breaks) from a young age, his first foray into Linux was in 2003 when, while blindly following the installation instructions in Sams Teach Yourself Red Hat Linux, he upgraded his dad’s Windows XP machine to Red Hat 9. This resulted in him getting his own computer, so it has been widely regarded as a good thing.

Show Notes

We had a lot of fun speaking with Ols! We discuss how a seemingly simple monitoring change ended up taking down the entire site. We also talk about chaos and resilience engineering, a topic Ols deeply cares about. We discuss how his team at Sky Betting and Gaming conducts fire drills (chaos engineering exercises) where they not only test the resiliency of their software systems but also their people systems. We walk through a recent example of a fire drill, how they have evolved over the past few years and the lessons learned in the process. Please enjoy this fun conversation with Ols!

Transcript

You can read the episode transcript here.

Links

• Ols on Twitter
• Ols on LinkedIn
• Ols’s personal website
• It’s just a monitoring change - Blog post by Ols where he wrote about the outage we discuss in the show