Apache Spark Streaming with Apache NiFi and Apache Kafka

Agenda

18:30 – 19:00: Food, mingling
19:00 PM – 19:45 PM: Lecture & demo
19:45 PM – 20:00 PM: Q&A

Apache Spark is a unified framework for big data analytics. Spark provides one integrated API for use by developers, data scientists, and analysts to perform diverse tasks that would have previously required separate processing engines such as batch analytics, stream processing and statistical modeling. Spark supports a wide range of popular languages including Python, R, Scala, SQL, and Java. Spark can read from diverse data sources and scale to thousands of nodes.

Overview Video: http://bit.ly/2cJ81pw

We will demo a Spark Streaming application ingesting simulated financial transactions in NiFi via a Kafka messaging broker. During the lecture we will cover Spark Streaming, NiFi, and Kafka in detail.

• Apache Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams.

• Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data.

• Apache Kafka is a high-throughput distributed publish-subscribe messaging system.

If you would like to replicate the demo, go ahead and setup one of the following:

• Hortonworks Sandbox (preconfigured HDP 2.4) on a Virtual Machine (VM) where you have full control of the environment. No data center, no cloud service, and no internet connection needed! http://hortonworks.com/products/sandbox/#install

or

• Hortonworks Sandbox (preconfigured HDP 2.4) in the cloud on Microsoft Azure. It’s FREE for the the first month, and there’s no need to download the VM! http://hortonworks.com/hadoop-tutorial/deploying-hortonworks-sandbox-on-microsoft-azure/

or

• Hortonworks Cloud on Amazon AWS for more control & pre-configured multi-node cluster deployments (slightly more advanced). Includes latest community builds, such as Spark 2.0: http://hortonworks.github.io/hdp-aws/

About the Speaker

Robert Hryniewicz has over 10 years of experience working on Machine Learning, AI, Robotics, cloud products and more. Currently he's a Data Scientist and Advocate at Hortonworks (a 100% open-source public Big Data company). Previously, Robert has been a principal consultant at TiVo, CTO at a Singularity Labs company, Sr. Engineer at Cisco, NASA, Concurrent et al. Robert has been developing in Apache Spark since 2014. As a consultant he developed several interesting products including a Graph Analytics platform, as well as multiple Machine Learning and IoT prototypes.

Robert’s interests range anywhere from distributed systems to advanced analytics, deep learning, NLP, general AI, robotics, VR, DNA Sequencing, personalized medicine, vertical farms, blockchain, and technologies related to promoting more open, democratic, peaceful, and cooperative societies.

He comes up with his best ideas when hiking close to home in Yosemite or when traveling to various remote regions around the world.

Contact Info

Email: rhryniewicz@hortonworks.com
Twitter: @RobH8z
LinkedIn: https://www.linkedin.com/in/roberthryniewicz