- Apache Flink Meetup Berlin X DataWorks Summit
This special Apache Flink Berlin Meetup will take place alongside DataWorks Summit 2018. Join us for an evening meetup, with snacks, drinks, and the latest and greatest on Apache Flink® at this wonderful location by the Spree. There will be a special discount code given for Dataworks Summit Berlin 2018. Program 6:00 pm Registration, Food & Drinks 6:30 pm Talk #1 by Aljoscha Krettek, data Artisans 7:00 pm Talk #2 by Ufuk Celebi, data Artisans 7:30 pm Mingle & Networking TALK #1 All your language are belong to us - Multi-language support on Apache Flink Speaker: Aljoscha Krettek Abstract: Data-stream/-batch processing and big data applications have been predominantly the domain of Java and Scala programmers. To overcome this, a recent effort in the Apache Beam and Apache Flink communities opened up stream processing to languages like Python and Go, including support for Python libraries for TensorFlow pipelines. This effort consisted of adding a "portability framework" to Beam, which allows developers to add new programming languages while reusing the existing hardened runtimes, and adding support for this portability framework to the Apache Flink runner. As a result, users can now run scalable stream processing pipelines with Python, Go across cloud services and on premise. The talk will show how to use the new APIs and libraries, as well as explain how Flink's existing stream/batch processing runtime interacts with the new languages. We present how we use Docker and Kubernetes to make the deployment and execution of those pipelines robust. TALK #2 Stateful Stream Processing in Containerland Speaker: Ufuk Celebi Abstract: TBA
- Apache Flink Meetup Berlin @SAP
We are thrilled to announce our third Apache Flink Berlin Meetup in 2017. A few days before the third edition of Flink Forward Berlin (https://berlin.flink-forward.org) kicks off, we’re organizing an evening meetup, with pretzels, beer, Mate, and the latest and greatest on Apache Flink® at our host SAP. This time we will have talks presented by data Artisans and ResearchGate. We hope you’ll join us to connect with each other ahead of the main event on September 11-13! Program (work in progress) 6:30 pm Registration, Food & Drinks 7:00 pm Introduction by SAP 7:15 pm Migration of a realtime stats product from Storm to Flink 8:00 pm Complex Event Processing with Flink: the state of FlinkCEP 8:30 pm Mingle & Networking Migration of a realtime stats product from Storm to Flink By Patrick Gunia, ResearchGate (https://www.researchgate.net/) Counting things might sound like a trivial thing to do. But counting things consistently at scale can create unique and difficult challenges. At ResearchGate we count things for different reasons. On the one hand we provide numbers to our members to give them insights about their scientific impact and reach. At the same time, we use numbers ourselves as a basis for data-driven product development. We continuously tune our statistics infrastructure to improve our platform, adapt to new business requirements or fix bugs. A milestone in this improvement process has been the strategic decision to move our stats infrastructure from Storm to Flink. This significantly reduced complexity and required resources, including decreasing the load on our database backend by more than 30%. We will discuss the challenges we’ve encountered and overcome on the way, including handling of state and the need for online and offline processing using streaming and batch processors on the same data. Patrick Gunia is an Engineering Manager at ResearchGate where he is responsible for the statistics infrastructure and the numbers it produces for internal and external usage. His special interest are streaming related technologies and how they can be integrated into an hybrid ecosystem consisting of real-time, near-realtime, and batch use cases. Complex Event Processing with Flink: the state of FlinkCEP By Kostas Kloudas, data Artisans (https://data-artisans.com/) Pattern matching over event streams is increasingly being employed in many areas including financial services and click stream analysis. Flink, as a true stream processing engine, emerges as a natural candidate for these usecases. In this talk, we will present FlinkCEP, a library for Complex Event Processing (CEP) based on Flink. At the conceptual level, we will see the different patterns the library can support, we will present the main building blocks we implemented to support them, and we will discuss possible future additions that will further enhance the coverage of the library. At the practical level, we will show how the integration of FlinkCEP with Flink allows the former to take advantage of Flink's rich ecosystem (e.g. connectors) and its stream processing capabilities, such as support for event-time processing, exactly-once state semantics, fault-tolerance, savepoints and high throughput. Kostas Kloudas is a Flink Committer, currently working with data Artisans to make Apache Flink® the best open-source stream processing engine and your data’s best friend. Before joining data Artisans, Kostas was a postdoctoral researcher at IST in Lisbon and even before that he obtained a PhD in Computer Science from INRIA (France). He obtained his engineering diploma from NTUA in Athens and his main research focus was in cloud storage and distributed processing.
- Apache Flink Meetup @idealo
We’re very excited to announce our second Apache Flink Meetup in 2017. Among others we will have talks presented by data Artisans. Join us for an evening of bagels, beer, and the latest and greatest on Apache Flink® at our host idealo (https://www.idealo.de/unternehmen/jobs/). Agenda: 7:00 pm Bagels, Beer, Networking 7:30 pm Introduction by idealo (https://www.idealo.de/unternehmen/jobs/) 7:40 pm Stateful Stream Processing with Apache Flink By Tzu-Li (Gordon) Tai, data Artisans (http://data-artisans.com) As Apache Flink continues to push the boundaries of stateful stream processing as an integral part of its past releases, increasing numbers of users are starting to realize the potential of stateful stream processing as a promising paradigm for robust and reactive data analytics as well as event-driven applications. This talk aims at covering the general idea and motivations of stateful stream processing, and how Flink enables it with its powerful set of state management features and programming APIs. In addition to that, we will also take a look at the recent advancements related to Flink's state management and large state handling that were driven by our team at data Artisans team in the latest version 1.3 (expected release by end of May / early June). 8:30 pm An introduction to Apache Beam via Nexmark, a benchmark suite for batch and streaming systems By Ismaël Mejía, Talend (https://www.talend.com/) Apache Beam is a unified programming model to express both batch and streaming use cases. By neatly separating properties of the data from run-time characteristics, Beam enables users to easily tune requirements around completeness and latency and run the same data processing job (pipeline) across multiple runtime environments including Apache Flink. In this talk we will detail some of the Apache Beam concepts by introducing Nexmark, a benchmarking suite implemented on Beam. We will discuss the properties that make Apache Beam the perfect tool to develop a benchmarking suite and how Nexmark bridges an existing gap in the Big Data benchmarking world. Also we’ll see how Nexmark serves as a rich integration test of the correct implementation of both the Beam runners (for systems like Apache Spark, Apache Flink and Apache Apex) and the new features on Beam.
- Apache Flink Meetup @ResearchGate
We’re very excited to announce our first Apache Flink Meetup in 2017. This time we will have talks presented by our host ResearchGate and by data Artisans. Join us for an evening of pizza, beer, and the latest and greatest on Apache Flink®. Agenda (work in progress): 1. Introduction by ResearchGate 2. Joining Infinity – Windowless Stream Processing with Flink By Sanjar Akhmedov, ResearchGate (https://www.researchgate.net/) The extensive set of high-level Flink primitives makes it easy to join windowed streams. However, use cases that don’t have windows can prove to be more complicated, making it necessary to leverage operator state and low-level primitives to manually implement a continuous join. This talk will focus on the anomalies that present themselves when performing streaming joins with infinite windows, and the problems encountered operating topologies that back user-facing data. We will describe the approach taken at ResearchGate to implement and maintain a consistent join result of change data capture streams. Sanjar Akhmedov is a software engineer at ResearchGate, with a focus on Flink and Hadoop stack. He is interested in distributed data processing, journaling systems and performance optimization. 3. Extending Flink's Streaming APIs By Kostas Kloudas, data Artisans (http://data-artisans.com/) As more and more organizations and individual users turn to Apache Flink for their streaming workloads, there is a bigger demand for additional functionality out-of-the-box. On one hand, there is demand for more low-level APIs that allow for more control, while on the other, users ask for more high-level additions that make the common cases easier to express. This talk will present the new concepts added to the Datastream API in Flink-1.2 and for the upcoming Flink-1.3 release that tried to consolidate the aforementioned goals. We will talk, among others, about the ProcessFunction, a new low level stream processing primitive that gives the user full control over how each event is processed and can register and react to timers, changes in the windowing logic that allow for more flexible windowing strategies, side outputs, and new features concerning the Flink connectors. Kostas Kloudas is a Flink Contributor, currently working with data Artisans to make Apache Flink® the best open-source stream processing engine and your data’s best friend. Before joining data Artisans, Kostas was a postdoctoral researcher at IST in Lisbon and even before that he obtained a PhD in Computer Science from INRIA (France), His main research focus was in cloud storage and distributed processing.
- Apache Flink Meetup @ data Artisans
data Artisans (http://www.data-artisans.com) is moving to a new office just across the canal! But before we do, we'd like to invite all of the Berlin Squirrels to Tempelhofer Ufer 17 one last time for an evening of pizza, beer, and the latest and greatest on Apache Flink®. Flink enthusiasts, come hang out with the data Artisans team and the local Flink community and ask dA's software engineers all of your pressing questions about Flink and stream processing. Agenda (work in progress): 1. Introduction and community update from data Artisans By Kostas Tzoumas & Robert Metzger, data Artisans Our CEO Kostas and co-founder Robert will talk about data Artisans and give an update on developments in Flink: new features, users, and the growing community. Kostas Tzoumas is PMC member of Apache Flink® and co-founder and CEO of data Artisans. Before founding data Artisans, Kostas was a postdoctoral researcher at TU Berlin and received a PhD in Computer Science from Aalborg University. Robert Metzger is Committer & PMC member member at the Apache Flink project and a co-founder and software engineer at dataArtisans. 2. A look at Apache Flink 1.2 By Stefan Richter, data Artisans As we are quickly moving towards the end of the year, version 1.2 of Apache Flink is already just around the corner. In this talk we give a full overview of the new release, which brings us dynamic job rescaling, integration with Apache Mesos, query able state, secure data access, and Kafka 0.10 support. We also take a closer technical look at some of the most anticipated features. In particular, we will present details on dynamic job rescaling and query able state that were driven by our team at data Artisans. With dynamic job rescaling, Flink 1.2 can now easily adjust operator parallelism to react to workload changes, while maintaining the fault tolerance guarantees and performance of previous versions. Queryable state is our first step to blur the boundaries between stream processors and databases with the potential to revolutionize the current state-of-the-art in big data. We present how to blend database functionality with Flink by providing live read-access to operator states in a running streaming pipeline. This talk concludes with a glimpse on the future roadmap beyond Flink 1.2 and how data Artisans plans to continue pushing the boundaries of stream processing. Stefan Richter is an Apache Flink® contributor and works as a software engineer at data Artisans. He has a PhD in Computer Science from Saarland University where he worked as researcher in the field of information systems. His research focus was on indexing, big data, and main memory databases. 3. Real-time analytics as a service at King By Gyula Fóra, King This talk introduces RBea, our scalable real-time analytics platform at King built on top of Apache Flink. The design goal of RBea is to make stream analytics easily accessible to game teams across King. RBea is powered by Apache Flink and uses the framework’s capabilities to it’s full potential in order to provide highly scalable stateful and windowed processing logic for the analytics applications. RBea provides a high-level scripting DSL that is more approachable to developers without stream-processing experience and uses code-generation to execute user-scripts efficiently at scale. In this talk I will cover the technical details of the RBea architecture and will also look at what real-time analytics brings to the table from the business perspective. If time permits I will also give some outlook on our future plans to generalise and further grow the platform. Gyula Fóra is a Data Warehouse Engineer in the Streaming Platform team at King, working hard on shaping the future of real-time data processing. This includes researching, developing and sharing awesome streaming technologies. Gyula grew up in Budapest where he first started working on distributed stream processing and later became a core contributor to the Apache Flink project.
- Apache Flink Meetup Berlin #14 at Zalando Tech
We are excited to invite you to our next Apache Flink Meetup, hosted by Zalando! Important: Please bring a valid ID for entry (as requested by our security staff). 1. Updates from the Community & A look at look at Apache Flink 1.1 By Stephan Ewen & Robert Metzger, data Artisans We will give an update on the developments in the Flink project: Flink Features, Users, and Community. After that, we look at some of the currently developed features (e.g., dynamic scaling, resource management, queryable state, Stream SQL) and what features are planned for the Flink 1.1 release. Stephan Ewen is Committer & PMC member in the Apache Flink project and co-founder and CTO of Data Artisans. Before founding dataArtisans, Stephan was leading the development of Flink since the early days of the project. Robert Metzger is Committer & PMC member member at the Apache Flink project and a co-founder and software engineer at dataArtisans. 2. Flink in Zalando's world of microservices By Javier Lopez & Mihail Vieru, Zalando In this talk we present Zalando's microservices architecture and introduce Saiki – our next generation data integration and distribution platform on AWS. We show why we chose Apache Flink to serve as our stream processing framework and describe how we employ it for our current use cases: business process monitoring and continuous ETL. We then have an outlook on future use cases.
- Apache Flink Meetup Berlin #13 Roadmap 2016 / Implementing BigPetStore
*Attention: New Location* We meet at the Event Kitchen @ Hubraum Join us at the 13th Apache Flink Meetup, drinks and sandwiches sponsored by data Artisans. The meet-up starts at 7 pm, the talks begin at 7:30 pm. 1. Flink Community Update & Roadmap 2016 By Stephan Ewen 2. Implementing BigPetStore - a blueprint for Flink users By Márton Balassi BigPetStore is a data generator application, which is part of the Apache BigTop project for testing and benchmarking data processing engines with implementations already available for MapReduce, Spark and now Flink. This talk is a practical walkthrough of the Flink implementation which uses a mixture of the DataSet, DataStream and Table APIs of Flink to express the application logic. For more infomation on the BigPetStore project please refer to Suneel Marthi's Flink Forward talk: http://flink-forward.org/?speaker=suneel-marthi About Márton Márton Balassi is a PMC member at Apache Flink and a researcher at the Hungarian Academy of Sciences. His main focus is real-time distributed data processing frameworks. Márton has been a speaker at Flink Forward, ApacheCon and numerous Big Data related meetups recently. Schedule 7:00 – 7:30 Sandwiches and Drinks 7:30 – 8:00 Flink Community Update / Roadmap[masked]:00 – 8:45 Implementing BigPetStore 8:45 – End Socializing and Drinks --------------- Bring your data After the talks, while having a drink, there's the opportunity to work together with Flink committers on an interesting data problem you're facing. Please contact Kostas Tzoumas at [masked] if you're interested in taking part in this! ---------------
- Flink Meetup #12: Year in Review / Modern Effective Mesos
*Attention: Meet-up was moved to December 16* Join us at the 12th Apache Flink Meetup, drinks and sandwiches sponsored by data Artisans. The meet-up starts at 7 pm, the talks begin at 7:30 pm. 1. Flink Community Update & Flink: The Year in Review By Robert Metzger 2. Modern Effective Mesos By Jörg Schad This talk will give an overview of the Apache Mesos project and its ecosystem. Then we will present a number of recent Mesos features which are relevant to both operators and framework writers including "IP per container", "Oversubscription", "Persistent Volumes", and "External Volumes". About Jörg Jörg Schad is a software engineer at Mesosphere in Hamburg where he works on the Apache Mesos project. Prior to this he worked on SAP Hana and in the Information Systems Group at Saarland University. His passions are distributed (database) systems, data analytics, and distributed algorithms and his speaking experience include various Meetups, international conferences, and lecture halls. Schedule 7:00 – 7:30 Sandwiches and Drinks 7:30 – 8:00 Flink Community Update / The Year in Review 8:00 – 8:45 Modern Effective Mesos 8:45 – End Socializing and Drinks --------------- Bring your data After the talks, while having a drink, there's the opportunity to work together with Flink committers on an interesting data problem you're facing. Please contact Kostas Tzoumas at [masked] if you're interested in taking part in this! ---------------
- Flink Forward 2015
Flink Forward 2015 is the first conference to bring together the Apache Flink developer and user community with 2 days of presentations and hands-on Flink training. Flink Forward 2015 is a conference targeted at Flink users, as well as data scientists interested in large-scale data analysis. The conference will consist of two parallel sessions: a presentation and a training session. Participants can expect: • technical presentations on Apache Flink by project committers, e.g., on system internals, as well as the project’s roadmap for future releases • use case presentations on Big Data projects using Apache Flink • presentations about related Big Data projects in the Apache ecosystem and beyond The parallel training session will feature two days of hands-on training workshops by Flink committers. No prior experience with the system is required. Is this the right conference for me? The primary target audience of Flink Forward is developers and data scientists working with Big Data tools and programming languages such as Java, Scala, and Python to make sense of streams of data. As the ecosystem and the community around Apache Flink is growing, Flink Forward offers a unique opportunity to meet the community, discuss the future directions of the project, and form future collaborations. Prior knowledge in Flink is not assumed, the workshop session is the perfect venue to get started with your first Flink jobs. Tickets include: • 2 Days of Talks and Workshops • All On-site Networking Events • Monday Evening Dinner • Monday Night Flink Fest • Meals During Flink Forward Tickets & further Info: http://flink-forward.org/ (http://flink-forward.org/?page_id=96)