Data Processing @SCALE

Almost every data backed product has the challenge of collecting, processing and presenting data and it has to be @scale, in the previous meetup we talked about collecting data and its challenges, come hear Iddo Rachlewski (Convertro’s VP of R&D) and Nader Ganayem (Senior DevOps Engineer) and share your deep thoughts on processing data @scale

This meetup will deeply cover the following topics:

1. How to design a robust and scalable data processing architecture

2. Why and how to use and utilize Labmda architecture

3. How to parse hundreds of data formats and integrate with #Hadoop

4. How to deal with out of order and belated data in the processing layer

5. How to manage jobs scheduling, workflows and dependencies

6. Monitoring, tracking and anomalies detection

7. What to do where: Collecting - Processing - Presentation layers

8. Hadoop-R integration and how it saved us 1MM$

9. #Devops Hadoop in the cloud - Yarn vs. Mesos vs. MR1

10. Ops Automation - Chef/Opsworks

11. EMR vs Cluster

#Hadoop #Vertica #R #Kafka #Elasticsearch #ELK #Anomaly-Detection #AWS

#Opsworks, #Chef, #Puppet, #Mesos, #Docker

Presenters:

Iddo Rachlewski, VP of R&D at Convertro (AOL) - https://il.linkedin.com/in/iddor

Nader Ganayem, Senior DevOps Engineer at Convertro (AOL) -

https://il.linkedin.com/in/nader-ganayem-655a0a3