Join us for our monthly gathering to discuss everything from incident response to cross-team collaboration to automation frameworks. Food provided. Please RSVP!
This month: DEVOPS HORROR STORIES
DevOps Mystery Stories #1: The Sinister Schema
On a bright spring morning, a high-profile website launch is underway. The company executives and assembled media watch their screens in feverish anticipation. An ocean away, on a dark and stormy night, a man in a black hooded cloak presses a button. Suddenly, a rogue exception appears! Will the launch be successful, or will the sinister schema strike again?
Matt Greensmith has been called many things, including "a devops." His favorite hobby is learning everything he can about other people's hobbies by watching YouTube videos (it's a very frugal hobby). When he's not watching YouTube videos, he's generally gluing infrastructure together with shell scripts, makefiles, and terraform. Matt is the Platform Engineering Lead at Dwelo, a smart home IoT startup, where he manages many thousands of tiny servers in peoples' homes.
Treehouse of Horror -- Ops Edition
Working in Ops and having horror stories almost seems to be a given. Allow me to tell you about some of the horrific shit I've seen in my years of keeping the systems up (or trying to, at least). One could probably rant for hours about such things, but I'll keep it to the three best/worst ones: that time I took down half of a 5000-person corporate campus with a single missing keyword, the cutover-that-just-wouldn't-end, and really great engineering on a really bad idea.
Mike Julian is the author of O'Reilly's Practical Monitoring, the Editor of the Monitoring Weekly newsletter, and an independent monitoring consultant at AsterLabs.io. Before embarking off as a consultant, he worked as an Ops Engineer for companies such as Taos Consulting, Peak Hosting, and Oak Ridge National Laboratory.