Yotpo Engineering #5 - Lessons for Scaling Smartly

Overview:

We're happy to invite you to our 5th meetup, hosted by our Full Stack Development Teams. This time, the sessions will focus on the challenges of scale and how we deal with them at Yotpo.
Please note: the lectures will be given in Hebrew.

Agenda:

18:00-18:30: Gathering
18:30-18:55: Talk #1: Stacked for Scale
18:55-19:20: Talk #2: Implementing a Scalable Queuing System Using Kafka
19:20-19:35: Break
19:35-20:00: Talk #3: Large Scale Data Processing Using Async Background Jobs
20:00+: Mingling

Abstracts:

18:30-18:55 - Stacked for Scale

Over the past few years, Yotpo’s customers have experienced tremendous growth and their needs have changed dramatically. These changes bring with them many product, technological, and organizational challenges. We’ve developed methodologies that support us in addressing those challenges.
In this lecture, we’ll share the results of an ongoing process: a curated playbook that gives every R&D department strategies for success at scale.

Yoni Biton,
Loyalty & Referrals Group Leader

18:55-19:20 - Implementing a Scalable Queuing System Using Kafka

Apache Kafka is an open-source stream-processing software platform. It aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
In this session, we’ll cover basic Kafka terminology: Topics, Groups, Partitions and Offsets. We will dive deeper into Kafka’s ACK policies, discuss their advantages and challenges, as well as how we built a queuing system using Kafka in order to support Yotpo’s architecture and vision of breaking the monolith. During the session, we’ll share real-life production use-cases and present the ecosystem and open-source tools we implemented at Yotpo to support message queuing requirements.

Yaniv Bronheim
Cloud Platform Group

19:30-20:00 - Large-Scale Data Processing Using Async Background Jobs

MapReduce has become a household name in the world of Big Data. As a framework, it allows for distributed processing of large inputs, as well as reducing their output and aggregating it.
As a growing SaaS company that provides various solutions for eCommerce brands, we are faced with the challenge of importing and exporting large quantities of data daily, and at a very large scale.
In this session, we’ll provide an insider’s look at how we implemented “Explodable Jobs,” a MapReduce-inspired framework that is capable of processing large quantities of data using Resque as an async background jobs solution.

David Laredo
Reviews & Ratings Group