Big data pipelines with high scale data processing in the cloud

Big data pipelines with high scale data processing in the cloud

Choose high accuracy or low latency or both?

Agenda:

18:30: Opening with food and drinks

19:00: Shipping YaaS logs with Apache Spark and Kafka
Dogukan Sonmez, hybris Software

19:30 Big Data - The Cloud Way
Alex Osterloh, Google

20:30: Q&A Session

Shipping YaaS logs with Apache Spark and Kafka

(Dogukan Sonmez hybris Software)

YaaS is hybris’ micro-service-based Business PaaS, and is integrated into hybris/SAP applications via a digital distribution channel. YaaS is Running on top of Cloud Foundry and all logs generated by the apps are being shipped via Apache Spark, running on top of Yarn and Using Apache Kafka for pipelining. At the end our pipeline ships logs to an elasticSearch cluster and presents it via Kibana to our developers. We are going to present challenges and technologies we are using for our logging pipeline.

http://photos2.meetupstatic.com/photos/event/c/0/9/6/600_444049302.jpeg

Big Data - The Cloud Way (Alex Osterloh Google)

We will look at some of the new technologies - that Google uses internally - that can now be used by anyone to ingest and process data, doing away with with lambda architectures. These include Pub/Sub, Dataflow, Dataproc and BigQuery. Will go through a live example (you can rebuild yourself, code on github) of building an event streaming pipeline using both bound and unbound data.

Speaker Bios

Alex Osterloh Google

In 2010 Alex Osterloh joined “Google for “Work” to bring Google technologies to the work force. He started with Enterprise Search solutions (e.g. Google Search Appliance, incl. Intranet, Web- and eCommerce Search) and is now part of the Google Cloud Platform team, focusing on embarrassingly parallel workloads, data analytics and visualization. Alex studied Computer Science and Statistics at Munich’s Ludwig-Maximilians University.

Dogukan Sonmez hybris Software

Dogukan works as a Senior Software Developer at hybris, with a main focus on big data applications and tools, He studied CSE (Computer Science and Engineering) and previously worked for Siemens Healthcare and Sony.

More to follow..

http://photos3.meetupstatic.com/photos/event/9/c/7/3/600_444940051.jpeg