Meetup @ QConNewYork - Survival of the Fittest - Streaming Architectures

NOTE DIFFERENT LOCATION: Marriot Marquis.

This month we are partnering with QCon NYC (https://qconnewyork.com/), and having a session in one of the evenings of the conference.

Michael Hansen, Principal Data Engineer @hbcdigital, will give us his QConn talk and then we will follow up with discussion and Q&A session with others QConn speakers from the streaming track.

Survival of the Fittest - Streaming Architectures

Abstract:

​“Perfect is the enemy of good” ​ ​ -​ ​Voltaire

On the journey through life, we learn and adapt via trial and error - software development is no different. We realize and accept that we won’t build the perfect solution the first time around, it takes many iterations. At Gilt.com, now part of HBC Digital, we started processing and streaming event data nearly 5 years ago. Our initial solution was dramatically different from our current solution - and will likely be different from our solution 5 years from now.

The Gilt.com banner, at HBC Digital, is in the business of flash sales, which makes for some interesting use cases in the world of streaming. We release new sales of top designer labels, at up to 70% off retail, on the web and our mobile app, every day at Noon and 9pm. Around the time of these releases, we experience volume spikes between 10X and 100X on our streams.

Numerous streaming frameworks, homemade, as well as, open source, did not pass the evolutionary tests. Frameworks come and go, ​so this talk is not about the “best” framework or platform to use, rather it’s about core principles that will stand the tests of streaming evolution. Also, this talk covers major potential pitfalls that you may stumble over on your path to streaming, as well as, how to avoid these. Finally, this talk will cover what the next evolutionary step in streaming at HBC Digital. ​

Speaker: Michael Hansen (https://www.linkedin.com/in/danishdatageek/)

https://secure.meetupstatic.com/photos/event/a/a/f/0/600_462103760.jpeg

Michael has two decades of technical and leadership experience in engineering big data, data warehouse, and data streaming systems, as well as, the full-stack environments, automation, and tooling surrounding these. Currently he is working as Principal Data Engineer at HBC Digital, responsible for all things related to data plumbing. He holds a Bachelor of Science in operations research and industrial engineering from UC Berkeley.

Agenda:

7:00 -7:30 - networking

7:30-8:15 - the talk

8:15- 9:00 - QA