Test In Production with Honeycomb - Horror Stories Edition


In honor of Halloween, we here at LaunchDarkly (together with Honeycomb.io!) are getting some speakers together to talk about their scariest stories of operational outages and other horrors. Testing in production can sound scary, but we want to hear some horror stories and some best practices for how to keeping our failures from becoming a disaster. What are the safeguards we can use, and how can we learn from each other to make our experimentation less spooktakular?

We will also hold a costume contest! Categories include:
Best (or worst?) Bug Costume
Best Team Costume
Punniest Costume
Best DIY Costume
Scariest (Outtage) Costume
Least Effort! (but still a costume)

There will be fab-u-less prizes! Come be judged by the one and only Charity Majors!


6:00 - 6:30 pm: Networking, food and drinks

6:30 - 7:30 pm: Speaker presentations

"The Time Our Provider Screwed Us" - Paul Biggar, Dark

Once upon a time, our hosted DB provider had a terrible security
incident, causing us to take down the entire product for 24 hours.
This is a story about the aftermath: the downtime, the incident
response, how we got back up, and how we communicated with customers.

"The Penny Glitch that Cost Big" - Marc Devens, BrandVerge

A change to a single line of code sent the prices of thousands of products on Amazon to a penny. Taking care of customers and focusing on engineering best practices allowed a company to survive and thrive after a "make or break" event.

"Two Tales" - Eric Pollmann, ClearBrain

First, a tale of a silent-but-deadly data push on a path to take Google Ads fully offline within 90 minutes, with no diagnosis in sight. Second, a zombie haunted pipeline that kept developers awake late into the night. ...each concluded with how we slayed the beast!

7:30 - 8:00 pm: Costume awards and open mic for more scary stories

8:00 - 8:30 pm: More networking and drinks

Please RSVP to make sure your name is on our list - we need to know how much food to order for everyone (don't worry we'll also have good veggie options)!