Conquer Big Data Challenges in Steaming, Security and Data Flow in IoT!

This is a past event

52 people went

Location image of event venue


David Callaghan will present Big Data challenges in IoT.

There are three major challenges with IoT that are not typically addressed, particularly at scale:

• 1. Bi-directional and point-to-point in addition to unidirectional streaming

• 2. Security, privacy and data protection at the "jagged edge"

• 3. Managing provenance in regulated environments

For many IoT startups, addressing these items would appear to be out of scope. However, larger regulated industries like health care, financial and retail consider these to be table stakes. A sensible approach would be to include an easy-to-use, open source platform that has the capability to scale in the very early stages of development. Apache NiFi fits this criteria.


Apache Nifi

Last November, the US National Security Agency open-sourced its Niagrafiles, or NiFi, data-flow software, to the Apache Software Foundation as Apache NiFi. Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Onyara, the company founded by some of the technology's NSA creators, was acquired by the Hadoop firm Hortonworks.


Open Enterprise Hadoop is an initiative to build Big Data platforms solely on open source components and on the open community. Hortonworks is currently the only major Hadoop platform distribution that is entirely composed of projects released under the rigor of the Apache Software Foundation.


The latest Hortonworks sandbox (2.3), includes a packaged version of NiFi, called Dataflow, available for experimentation, together with documentation, implementation tips and techniques.

Bio: Dave Callaghan - Sparks Ignite - Modern Data Innovator

Dave brings twenty years of development experience to today's modern data environments. He is certified in and works extensively and contributed to Hadoop and he has extensively used unstructured and semi-structured data in HBase, Cassandra and MongoDB. He implemented Big Fast Data solutions using Storm, Spark and Kafka streaming into both HBase and Hive using both Java and Scala. He is working with and contributing to the new data governance initiative platform applications like Falcon and Ranger and Atlas.