Storm at Spotify

This time we will visit Spotify HQ in New York City to learn about how they use Storm.

In this talk, Spotify engineer Neville Li will share their experience building real-time features with Storm and Kafka, including recommendation, social, data visualization and ads targeting. We will cover topics such as architecture, production integration, and best practices.

The commercial music streaming service Spotify was launched in 2008 and since then is has registered over 24 million active users of which 6 million are paying users. They have 3.7 million Facebook fans. It has over 20 million songs online and every day 20.000 new songs are added to the database. Users created over 1 billion playlists and over $ 500 million has been paid out to rights holders since the launch of Spotify. It may be clear that without big data techniques and tools used, Spotify would not be able to exist.

Spotify is a data-driven company, meaning that data is used in almost any part of the organization. The numbers confirm this: Spotify users create 600 Gigabyte of data per day and 150 Gigabyte of data per day via different services. Every day 4 Terabyte of data is generated in Hadoop, a 700-node cluster running over 2.000 jobs per day. They currently have 28 Petabytes of storage, spread out over 4 data centres across the world. This is the first time they will be talking about their deployment and use cases for Storm.

Neville Li (@sinisa_lyh (https://twitter.com/sinisa_lyh)) is a Software Engineer at Spotify, where he has been crunching data since 2011 and has introduced Storm, Scalding, and Spark to Spotify growing data ecosystem.

http://photos3.meetupstatic.com/photos/event/5/0/2/a/event_338480522.jpeg

As always, we will have book raffle sponsored by O'Reilly.

Food and drinks will be provided by Spotify.

Agenda:

6:30 - Arrive to Spotify, meet other members
6:45 - Books giveaway
7:00 - Storm at Spotify
8:00 - Q&A
8:15 - Open Discussion, Networking

Location:

Spotify 45 West 18th St
7th floor
New York, NY

Our Sponsors:

DataTorrent (http://www.datatorrent.com/)

DataTorrent is the most powerful real-time computation platform.

NoSQL Weekly (http://www.nosqlweekly.com/)

A free weekly newsletter featuring curated news, articles, new releases, jobs (http://jobs.nosqlweekly.com/) etc related to NoSQL.