János Csorvási (Meltwater): To a billion messages and beyond — A story about the evolution of Meltwater’s log ingestion pipeline
Logs are great, huh? Well... not really. Unless you need them. But when you need them, you REALLY need them. Especially when you’re debugging and set level to TRACE, in a critical loop in your data processing pipeline, spewing out gigabytes of data in mere minutes. What happens then?
Join us to learn about how Meltwater, a leading media monitoring company, evolved its log ingestion pipeline over the years, from indexing 50 events per second to more than[masked] events per second (2 billion events per day) today. This is a story about evolution, data pipelines, bugfixes leading to capacity problems, scalability challenges, datacenter-to-cloud migration, massive data volume increases, war stories and last but not least, how our team is preparing to ingest 100 times this.
Technologies covered include Elasticsearch, Logstash, Kibana, Redis, Kafka, AWS S3, AWS Lambda, Docker, Mesos/Marathon and Kubernetes.
Gergely Brautigam (Cheppers): It's not illegal to cook infrastructure
aka. How not to break things bad!
It's not illegal to bake infrastructure, in fact, it's encouraged. Baking it means that you have a receipt. If you have a receipt, you can repeat said baking. The more often you repeat it, the better you get at it. You refine your receipt until perfection, at which point you put it into a golden frame and make it your families heirloom. Now, what if I told you that you can automate the whole process and feed your receipt to a cooker that will produce the same cake: Every. Single. Time. Meet Furnace and AWS CloudFormations.
As usual, the schedule:
* 18:00: Doors open
* 18:30: First presenter
* 19:15: Office visit + taking pictures
* 19:30: Second presenter
* 20:00: pizza & free discussion
* 21:00: EOF
This meetup event is English speaking! Pizza and beers are sponsored by Cloudera.
Some things to remember:
• We always welcome talk proposals! Please submit your talk on Slack (https://join.slack.com/t/budapestdevopsmeetup/shared_invite/enQtMjk2OTgzODc2NjQyLWU3ZWM5OGZiYzljOTUwNmJkNTE3OGM0N2Q5YmYwOWM0Nzg3YzMxYWQwNzFjZmY5ZmZlYzkyN2NlY2MyN2YxNjM) or via this form: https://goo.gl/6OEBRJ .
• If you have something you'd like to hear about, please let us know via Slack, or this other form: https://goo.gl/DVjHju