Streaming app with <2ms latency in Hadoop/Apex; and Ingestion from Nifi

Agenda - This event has two talks; the first talk with cover a fully fault tolerant distributed native Hadoop application with under 2ms latency. The second talk covers ingestion and egression of data into/from Hadoop using Nifi. The Nifi connectors are part of Apache Malhar library.

5:45pm - Food and Drinks

6:15pm - Toppling the mainframe: Enterprise-grade streaming under 2ms on Hadoop - Ilya Ganelin (Capital One Data Innovation Lab)

7:00pm - Q&A and break

7:15pm - Integrating Apex and Apache NiFi by Bryan Bende (Hortonworks)

8:00pm - Q&A, break and wrap-up

Sponsored by Ampool

Abstract for - Toppling the mainframe: Enterprise-grade streaming under 2ms on Hadoop:

These days everyone is excited about big data and fast data. Capital One has embraced this new generation of technology with open arms. However, “There ain’t no such thing as a free lunch.”

For many years, there’s been a very real battle around the standard operating model of software. Tech giants like Oracle and IBM have traditionally built massively expensive enterprise-ready products, while the open source community provides free, albeit usually inferior, software. For a product to be enterprise-ready, it must guarantee complete reliability alongside performance and flexibility. We had not yet seen open source succeed in the realm of distributed stream computing.

Capital One set out to find whether we could build or find enterprise-ready technology in the open source world to tackle difficult streaming problems that also provides equivalent performance, durability, and availability as a mainframe computer. Ilya Ganelin details Capital One’s attempt to answer this question in a rigorous and complete way, not just by making a prototype or discovering exciting new tools, but by creating an open source-based, enterprise-ready product that can transparently replace an enormously expensive proprietary solution. Ilya presents Capital One’s novel solution for real-time decisioning and computations on Apache Apex.

Topics include:

• A detailed dive into the business requirements of a new real-time decisioning platform for model building, feature computation, and model scoring.

• A survey and analysis of the leading open source technologies for stream processing and what tradeoffs Capital One considered when selecting their technology stack.

• Capital One’s solution, based on Apache Apex, which provides unparalleled performance on Hadoop and meets the stringent performance, scalability, and durability requirements necessary for enterprise-grade decision making

Bio: Ilya Ganelin is a roboticist turned data engineer. After a few years building self-discovering robots at the University of Michigan and another few years working on embedded DSP software with cell phones and radios at Boeing, he landed in the world of big data at the Capital One Data Innovation Lab. Ilya is an active contributor to the core components of Apache Spark and a committer to Apache Apex with the goal of learning what it takes to build a next-generation distributed computing platform. Ilya is an avid bread maker, cook, skier, and race-car driver.

Abstract for - Integrating Apex and Apache NiFi; Ingesting and Egressing data from/to Nifi to Hadoop

In this talk we will give an overview of Apache NiFi and how it can be used to bring data to a native Hadoop analytic platform like Apache Apex. Specifically we’ll discuss the details of the NiFi input and output operators that were developed for the Apache Apex Malhar library, and talk through use-cases for integrating the two technologies.

Bio: Bryan Bende is a Member of the Technical Staff at Hortonworks where he develops dataflow capabilities around the core framework of Apache NiFi, and has over ten years of experience developing enterprise software solutions. Bryan received a B.S. in Computer Science from the University of Maryland at College Park, and a M.S. in Computer Science from John Hopkins University.