Monitoring and Error Reporting in High-Scale Production Systems

Monitoring production systems and understanding them requires detailed information, reported when needed. High-scale production systems call for a scalable approach. Join us to hear how Taboola and Wix Engineers tackle these challenges.

18:00-18:30 Gathering and Refreshments

18:30-19:15
Monitoring at Scale, Gali Katz

Taboola is the world’s largest recommendation platform. As such, we deal with massive scale (1.4B users/month) and extremely tight SLAs in a very competitive market. In order to do so, we monitor thousands of servers in numerous data centers around the world and process over 100 million metrics per minute. These metrics are an integral part of our unique ecosystem and critical for maintaining the business life and functioning.

We built an impressive metrics architecture, combining different time series DBs using various techniques, to maintain a huge amount of data. Starting with pre-processing metrics with high cardinality, through data transformation, aggregation, sampling, and finally, using our own creative heuristics for smart analysis.

In this talk Gali will share the journey we’ve made towards monitoring at scale at Taboola.

Gali is a Full Stack Developer in the Infrastructure Engineering R&D group at Taboola, and a Ph.D student in Cognitive Science at Ben Gurion University (https://www.linkedin.com/in/galikatz/)

19:15-20:00
Error-reporting in Modern FED Ecosystems, Ziv Levy

Do you use an error reporting system in your software? How do you distinguish between errors and filter them? How do you run your tests without throwing tons of errors to that system? How do you even test error reporting?
My talk is about error-reporting in modern FED eco-systems. I’ll present our approach of how to scope and differentiate between errors in “multi-app” systems, and how to conduct testing conveniently and with no hacks.
I truly believe that the story about how we solved it, would benefit and inspire others.

Ziv Levy is a Software Engineer at Wix.com (https://www.linkedin.com/in/levyziv/)