• Kafka on Kubernetes: Does it really have to be “The Hard Way”?

    Macquarie Business Banking

    Details Join us for our next Sydney Apache Kafka meetup on October 17th from 6:00pm hosted by Macquarie Group. The agenda, venue and speaker information can be found below. See you there! 6:00pm - Entry, networking, food/drink 6:30pm - Simon Aubury, ThoughtWorks 7:00pm - Viktor Gamov, Confluent 8:00pm - Additional Networking and Q&A 8:30pm - Doors close Speaker: Speaker: Viktor Gamov Title: Kafka on Kubernetes: Does it really have to be “The Hard Way”? Abstract: When it comes to choosing a distributed streaming platform for real-time data pipelines, everyone knows the answer: Apache Kafka! And when it comes to deploying applications at scale without needing to integrate different pieces of infrastructure yourself, the answer nowadays is increasingly Kubernetes. However, with all great things, the devil is truly in the details. While Kubernetes does provide all the building blocks that are needed, a lot of thought is required to truly create an enterprise-grade Kafka platform that can be used in production. In this technical deep dive, Michael and Viktor will go through challenges and pitfalls of managing Kafka on Kubernetes as well as the goals and lessons learned from the development of the Confluent Operator for Kubernetes. Bio: Viktor Gamov is a Developer Advocate at Confluent, the company that makes an event streaming platform based on Apache Kafka. Back in his consultancy days, Viktor developed comprehensive expertise in building enterprise application architectures using open source technologies. He enjoys helping architects and developers to design and develop low latency, scalable and highly available distributed systems. He is a professional conference speaker on distributed systems, streaming data, JVM and DevOps topics, and is regular on events including JavaOne, Devoxx, OSCON, QCon, and others. He co-authored O’Reilly’s «Enterprise Web Development.» He blogs at http://gamov.io and co-hosts DevRelRad.io. Follow Viktor on Twitter @gamussa, where he posts there about gym life, food, open source, and, of course, Kafka and Confluent! --- Speaker: Simon Aubury Title : Machine learning & Kafka KSQL stream processing Continuous household power monitoring with a trained model to alert on unexpected power usage for time of day. Building a ML UDF for KSQL and using Apache Kafka to alert me via our Google Home when I’ve left the iron running . Bio : Simon Aubury is a Principal Data Engineer at ThoughtWorks. He loves building enterprise streaming applications and has worked on Kafka platforms for banks, insurance & real-time vehicle tracking. If you would like to speak or host our next event please let us know! [masked] NOTE: We are unable to cater for any attendees under the age of 18. Please do not sign up for this event if you are under 18.

    1
  • Nick Dearden and KSQL

    Macquarie Business Banking

    Join us for our next Sydney Apache Kafka meetup on August 13th from 6:00pm hosted by Macquarie Group. The agenda, venue and speaker information can be found below. See you there! 6:00pm - Entry, networking, food/drink 6:30pm - Nick Dearden, Confluent 7:30pm - Simon Aubury, ThoughtWorks 8:00pm - Additional Networking and Q&A 8:30pm - Doors close Speaker: Nick Dearden Director of Engineering, Confluent Bio: Nick is one of the masterminds and co-creators of KSQL. Nick is also a technology and product leader at Confluent, where he enjoys leveraging many years of experience in the world of data and analytic systems to help design and explain the power of a streaming platform for every business. Prior to Confluent, he led the data platform group for a leading online real-estate seller and was chief architect for a cloud-based financial analytics platform. His early career stretches all the way back through multiple data warehouse and business intelligence adventures to the green-screen days of mainframe banking systems. Title: Zen and the Art of Streaming Joins—The What, When and Why One of the most powerful capabilities of both KSQL and the Kafka Streams library is that they allow us to easily express multiple “types” of join over continuous streams of data, and then have those joins be executed in distributed fashion by a self-organizing group of machines—but how many of us really understand the intrinsic qualities of a RIGHT OUTER STREAM-STREAM JOIN, SPAN(5 MINUTES, 2 MINUTES)? And what happens when that data can arrive late or out of order? In this talk we will explore the available streaming join options, covering common uses and examples for each, how to decide between them and illustrations of what’s really happening under the covers in the face of real-world data. --- Speaker: Simon Aubury Principal Data Engineer, ThoughtWorks Bio : Simon Aubury is a Principal Data Engineer at ThoughtWorks. He loves building enterprise streaming applications and has worked on Kafka platforms for banks, insurance & real-time vehicle tracking. You can find his KSQL projects for monitoring cats and home power usage on github, and his tutorials on Udemy. Title : Did I beat Ben? Building a live race tracking platform with Kafka streaming & MQTT. Simon presenting a short talk on tracking runner locations mapping their location & race statistics using KSQL & displaying live visualisations with Kibana. How to track friends and family when they’re silly enough to run 21km If you would like to speak or host our next event please let us know! [masked] NOTE: We are unable to cater for any attendees under the age of 18. Please do not sign up for this event if you are under 18.

    6
  • Developing Contextual, Event-Driven Applications with KSQL and Kafka

    Macquarie Business Banking

    **** Please RSVP with your full name and bring photo ID to ensure entrance to the building as per security rules. **** ----- Agenda: 6:00pm: Doors open 6:00-6:30pm Pizza, drinks and networking 6:30pm-7:15pm David Petersen, Systems Engineer, APAC | Confluent 7:15pm-7:45pm Q&A with Hans Jespersen, Vice President, WW Systems Engineering, Confluent 7:45-8:00pm Additional networking ---- Speaker: David Petersen, Systems Engineer, APAC | Confluent Bio: David is helping to shape the modern view of systems, introducing new ways of thinking, new patterns of Internet scale systems. He is a key strategic advisor helping to bring real time event systems to the forefront and shape opinions of how distributed systems should be built. David leads Confluent's APAC systems strategy and implementation with new and existing clients along with the technical implementation across the region. Title of Talk: Developing Contextual, Event-Driven Applications with KSQL and Kafka Abstract: Unordered, unbounded and massive datasets are increasingly common in day-to-day business. Applications that combine these real-time data streams with historical context brings a massive advantage to solving very difficult tasks. A real time Kafka platform moves your data up the stack, closer to the heart of your business, allowing you to build scalable, mission-critical services by quickly deploying SQL-like queries in a serverless pattern. KSQL is the streaming SQL engine for Apache Kafka. KSQL lowers the entry bar to the world of stream processing, providing a simple and completely interactive SQL interface for processing data in Kafka. KSQL (like Kafka) is open-source, distributed, scalable, and reliable. This talk will show you how to take advantage of this power by adopting new ways of using and looking at data by highlighting key use cases for real time data, and stream processing with KSQL: Real time analytics, security and anomaly detection, real time ETL / data integration, Internet of Things, application development, and deploying Machine Learning models with KSQL. ----- Don't forget to join our Community Slack Team! https://launchpass.com/confluentcommunity If you would like to speak or host our next event please let us know! [masked] NOTE: We are unable to cater for any attendees under the age of 18. Please do not sign up for this event if you are under 18.

    1
  • Tim Berglund talks KSQL

    Macquarie Bank Headquarters

    Join us for our next Apache Kafka® meetup on September 26th hosted by Macquarie. The agenda, venue and speaker information can be found below. See you there! ----- Agenda: 6:00pm: Doors open: Pizza & Drinks 6:30pm: Tim Berglund, Confluent 7:15 - 7:30 - Drinks, Networking and additional Q&A ------ Speaker: Tim Berglund, Confluent Bio: Tim Berglund is a teacher, author, and technology leader with Confluent, where he serves as the Senior Director of Developer Experience. He can frequently be found at speaking at conferences in the United States and all over the world. He is the co-presenter of various O’Reilly training videos on topics ranging from Git to Distributed Systems, and is the author of Gradle Beyond the Basics. He tweets as @tlberglund, blogs very occasionally at http://timberglund.com (http://timberglund.com/), is the co-host of the http://devrelrad.io (http://devrelrad.io/) podcast, and lives in Littleton, CO, USA with the wife of his youth and their youngest child, the other two having mostly grown up. Title: Processing Streaming Data with KSQL Abstract: Apache Kafka is a de facto standard streaming data processing platform, being widely deployed as a messaging system, and having a robust data integration framework (Kafka Connect) and stream processing API (Kafka Streams) to meet the needs that common attend real-time message processing. But there’s more! Kafka now offers KSQL, a declarative, SQL-like stream processing language that lets you define powerful stream-processing applications easily. What once took some moderately sophisticated Java code can now be done at the command line with a familiar and eminently approachable syntax. Come to this talk for an overview of KSQL with live coding on live streaming data. ------ Don't forget to join our Community Slack Team! https://launchpass.com/confluentcommunity If you would like to speak or host our next event please let us know! [masked] NOTE: We are unable to cater for any attendees under the age of 18. Please do not sign up for this event if you are under 18.

  • 5 Steps to Build streaming systems- Neha Narkhede ( CTO at Confluent)

    Neha Narkhede is co-founder and CTO at Confluent, a company backing the popular Apache Kafka messaging system. Prior to founding Confluent, Neha led streams infrastructure at LinkedIn, where she was responsible for LinkedIn’s streaming infrastructure built on top of Apache Kafka and Apache Samza. She is one of the initial authors of Apache Kafka and a committer and PMC member on the project. Agenda ======== Neha will cover a five-step adoption journey we observe organisations going through as they build streaming systems. First, the organisation becomes aware of streaming as an architectural paradigm and builds a pilot system. After that pilot succeeds, streaming goes live with the first production system. Third, the small production system spreads to an mission-critical application. Fourth, as the organisation gains competence with the new platform and comes to depend more and more on the advantages conferred by a streaming platform, that mission-critical application is expanded to a global use case, where streaming data is produced and consumed across geographies. Finally, the organisation learns that a streaming platform is the proper core on which to layer all of its applications and services, and the “central nervous system” metaphor of comprehensive streaming adoption finally becomes true. Streaming adoption, having begun with an idea and a pilot project, has finally transformed a business and its entire information architecture.

    21
  • Kafka Deployment & Management in Prod

    1 Shelley Street

    Join us for our next Sydney Apache Kafka meetup on Jun 12th from 6:00pm hosted by Macquarie Group. The agenda, venue and speaker information can be found below. See you there! ----- Agenda: 6:00pm: Doors open 6:30pm - 7:30pm: Ben Slater, Instaclustr 7:30 - 8:00 - Pizza, Drinks, Networking and additional Q&A Title: Tips and Tricks for Automating the deployment and management of Apache Kafka Abstract: Instaclustr has recently extend its managed service for Apache Cassandra to include support for Apache Kafka. This presentation will walk through the key challenges and lessons learned in extending Instaclustr’s provisioning and management system for Kafka. Speaker: Ben Slater, CPO, Instaclustr Bio: As Chief Product Officer, Ben is charged with steering Instaclustr’s development roadmap and overseeing the product engineering, production support, open source and consulting teams. Ben has over 20 years experience in systems development including previously as lead architect for the product that is now Oracle Policy Automation and over 10 years as a solution architect and project manager for Accenture. He has extensive experience in managing development teams and implementing quality controlled engineering practices.

    4
  • Stream All Things with Kafka and KSQL

    1 Shelley Street

    Join us for our next Sydney Apache Kafka meetup on May 15th from 6:00pm hosted by Macquarie Group. The agenda, venue and speaker information can be found below. See you there! ----- Agenda: 6:00pm: Doors open 6:15pm - 7:00pm: Nick Dearden, Confluent 7:00 - 8:00 - Pizza, Drinks, Networking and additional Q&A Speaker: Nick Dearden, Confluent Bio: Nick is a technology and product leader at Confluent, where he enjoys leveraging many years of experience in the world of data and analytic systems to help design and explain the power of a streaming platform for every business. Prior to Confluent, he led the data platform group for a leading online real-estate seller and was chief architect for a cloud-based financial analytics platform. His early career stretches all the way back through multiple data warehouse and business intelligence adventures to the green-screen days of mainframe banking systems. Title: Streaming ETL with Apache Kafka and KSQL Abstract: In this talk, we'll see how easy it is to capture a stream of data changes in real-time from a database such as MySQL into Kafka using the Kafka Connect framework and then use KSQL to filter, aggregate and join it to other data, and finally stream the results from Kafka out into multiple targets such as Elasticsearch and MySQL. All of this can be accomplished without a single line of Java code!

    4
  • Kafka, KSQL

    Level 26

    Please use the eventbrite page to register for this event. https://www.eventbrite.com.au/e/ksql-for-apache-kafka-presented-by-nick-dearden-from-confluent-tickets-39679226635 Agenda: 6:00pm: Doors open 6:00pm - 6:30pm: Networking, Pizza, and Drinks 6:30pm - 7:15pm: Presentation #1: KSQL, Nick Dearden, Confluent 7:15pm - 8:00pm: Presentation #2: How to use Apache Kafka to transform a batch pipeline into a real-time one?, Stephane Maarek, Simple Machines 8:00pm - 8:30pm: Additional Q&A and Networking About the Speaker: Nick Dearden Nick is a technology and product leader at Confluent, where he enjoys leveraging many years of experience in the world of data and analytic systems to help design and explain the power of a streaming platform for every business. Prior to Confluent, he led the data platform group for a leading online real-estate seller and was chief architect for a cloud-based financial analytics platform. His early career stretches all the way back through multiple data warehouse and business intelligence adventures to the green-screen days of mainframe banking systems. KSQL KSQL is an open source streaming SQL engine for Apache Kafka. Come hear how KSQL makes it easy to get started with a wide-range of stream processing problems such as real-time ETL, sessionization, monitoring and alerting, or fraud detection. We'll cover both how to get started with KSQL and some under-the-hood details of how it all works. About the Speaker: Stephane Maarek Stephane is an engineer impassioned by all things relating to big and fast data. Stephane is an Apache Kafka subject matter expert at Simple Machines, has authored several blog posts in partnership with Confluent, and is a best-selling instructor on Udemy for his courses on Apache Kafka, Apache NiFi and AWS Lambda. How to use Apache Kafka to transform a batch pipeline into a real-time one? Apache Kafka has real-time capability and everyone knows that! The real challenge facing engineers comes from re-designing the existing data pipelines from batch to real-time. In this talk, we will do a case study on how to build an end-to-end real-time data pipeline by building four micro-services on top of Apache Kafka. It will give you insights into the Kafka Producer API, Avro and the Confluent Schema Registry, the Kafka Streams High-Level DSL, and Kafka Connect Sinks.

    3
  • Kafka & Kafka Streams

    Location visible to members

    As organisations move towards realtime and fast data solutions we will find more edge computing architecture supported by event streaming and event sourcing patterns. We are excited to be hosting the first Kafka meetup at Macquarie Headquarters on the 5th of April from 6:15PM. Jeremy Custenborder will be giving an e-talk on Introduction to Kafka and Kafka Streams. Jeremy is a systems engineer with Confluent. Kafka Streams is a library for building distributed stream processing applications using Apache Kafka. Rajay Rai from Macquarie bank will speak about Kafka use in Macquarie. Rajay is the head of applied innovation in Banking and Financial Services within Macquarie Group. Come and join us for pizzas and beers. Please be aware that it MANDATORY for all attendees to register and bring an ID for security check. Please be aware that friends (+1 also need registration).

    1