Tuning and Productionizing Storm

Hey guys,

 

It's been a while, let's get together again to talk Storm!

We thought we would focus this meetup more on tuning and productionizing Storm.

Nathan will give a talk on tuning Storm, and I'll give a talk on some other aspects of productionizing Storm, like load testing.

 

 

Pizza and drinks will be served, first come first serve.

 

NOTE: The doors downstairs will be locked at 7pm, please come in before then.

 

There will only be seating space for 30.  The rest will be stand up.  Please bring your own chairs if needed.

----

Webex info

-----

Hello ,

Topic: Storm Meetup IV
Date: Wednesday, November 14, 2012
Time: 6:00 pm, Pacific Standard Time (San Francisco, GMT-08:00)
Meeting Number:[masked]
Meeting Password: stream


-------------------------------------------------------
To join the online meeting (Now from mobile devices!)
-------------------------------------------------------
1. Go to https://ciscosales.webex.com/ciscosales/j.php?ED=210056977&UID=1915593437&PW=NYTkxOTRkNmEy&RT=MiM0
2. Enter your name and email address.
3. Enter the meeting password: stream
4. Click "Join Now".

To view in other time zones or languages, please click the link:
https://ciscosales.webex.com/ciscosales/j.php?ED=210056977&UID=1915593437&PW=NYTkxOTRkNmEy&ORT=MiM0

----------------------------------------------------------------
ALERT:Toll-Free Dial Restrictions for (408) and (919) Area Codes
----------------------------------------------------------------

The affected toll free numbers are: (866)[masked] for the San Jose/Milpitas area and (866)[masked] for the RTP area.

Please dial the local access number for your area from the list below:
- San Jose/Milpitas (408) area:[masked]
- RTP (919) area:[masked]

-------------------------------------------------------
To join the teleconference only
-------------------------------------------------------
1. Dial into Cisco WebEx (view all Global Access Numbers at
http://cisco.com/en/US/about/doing_business/conferencing/index.html
2. Follow the prompts to enter the Meeting Number (listed above) or Access Code followed by the # sign.

San Jose, CA: [masked] RTP: [masked]

US/Canada: [masked] United Kingdom: [masked]

India: [masked] Germany: [masked]

Japan: [masked] China: [masked]

-------------------------------------------------------
For assistance
-------------------------------------------------------
1. Go to https://ciscosales.webex.com/ciscosales/mc
2. On the left navigation bar, click "Support".

You can contact me at:
[masked]
[masked]

To update this meeting to your calendar program (for example Microsoft Outlook), click this link:
https://ciscosales.webex.com/ciscosales/j.php?ED=210056977&UID=1915593437&ICS=MRS2&LD=1&RD=2&ST=1&SHA2=R2P9motEsg7GaAzeY6oj7Ik/mlrLIylu61PQttOyCkU=&RT=MiM0


WebEx will automatically setup Meeting Manager for Windows the first time you join a meeting. To save time, you can setup prior to the meeting by clicking this link:
https://ciscosales.webex.com/ciscosales/meetingcenter/mcsetup.php


The playback of UCF (Universal Communications Format) rich media files requires appropriate players. To view this type of rich media files in the meeting, please check whether you have the players installed on your computer by going to https://ciscosales.webex.com/ciscosales/systemdiagnosis.php.




http://www.webex.com

CCP:[masked]x[masked]#


IMPORTANT NOTICE: This WebEx service includes a feature that allows audio and any documents and other materials exchanged or viewed during the session to be recorded. By joining this session, you automatically consent to such recordings. If you do not consent to the recording, discuss your concerns with the meeting host prior to the start of the recording or do not join the session. Please note that any such recordings may be subject to discovery in the event of litigation.

Join or login to comment.

  • eric c.

    I think it was helpful as an overview and set of specific challenges that are being worked on as part of eminent releases (i.e. 0.8.2). What I think was disappointing and surprising was the lack of hard statistics / metrics on running storm clusters and some interesting deployment / scaling battle scales. Would have made the session a lot more interesting to hear more of the tuning aspects of a high scale system.

    Another area that popped up a decent amount in the Q&A was why storm. There are a more interesting stream processed based systems popping up and I think many folks in the audience were interested in evaluating storm vs some other framework. Know this was not the purpose of the talk. But it would have been interesting to see compare / contrast of storm from a deployment perspective with other emerging stream processing stacks. Maybe topic for next meetup ...

    November 16, 2012

  • Shaji Ravindra N.

    Very Good

    November 15, 2012

  • Andrey

    Nathan gave crisp explanation of the most complex parts of Storm

    November 14, 2012

  • A former member
    A former member

    excellent talks

    November 14, 2012

  • Shingo O.

    Thank you Nathan! Good Talk today!
    Will today's slide be shared on the web??

    November 14, 2012

  • Evan C.

    By the way, please plan on coming into the building before 7pm, as after 7pm both the downstairs garage and the front main doors will lock. Thanks!

    November 13, 2012

  • Evan C.

    By the way the WebEx info is up.

    November 13, 2012

  • Jack M.

    Evan/Organizers: Is there a way you can upload a webcast of this event, please? Getting a production deployment "right" with storm is really tough, so I think this could be a very valuable resource for folks using the tech in the future as well as for people who can't attend on Wednesday. We use ustream.tv to archive our hadoop meetups in Los Angeles, fyi.

    To add a talking point: I would like to know how to gauge how many supervisors / workers / bolt parallelism / to have per storm node.

    Thanks!!

    November 12, 2012

    • Evan C.

      We will be recording it, and I'll see if someone can help set up WebEx or ustream.tv. To your point, I think Nathan's talk will answer many of your questions, but we keep it pretty simple: 1 supervisor & 1 worker per node, makes debugging much easier.

      November 13, 2012

    • Matt M.

      I'd be interested in a recording or remote attendance as well.

      November 13, 2012

  • Mike H.

    Some talking points I'd find valuable:

    * Best methods for making Nimbus nodes Highly-Available on EC2.

    * How to extract internal Storm metrics so that they can be pushed to external metrics/monitoring systems (CloudWatch, Graphite, Librato Metrics, etc). Also, what do the metrics mean and which ones are most important for tracking the health of Storm.

    * Real-world examples of automated scaling of Storm clusters. Maybe using EC2 Auto-Scaling-Groups and dynamic rebalancing?

    * How to deploy Storm -- examples of CI integration, Hubot automation, etc.

    * As a Kafka user, best practices or anti-patterns when using Storm+Kafka.

    October 23, 2012

  • Evan C.

    Now if anyone has questions regarding Storm tuning or productionizing, please feel free to post it before we get started.

    Thanks!

    October 23, 2012

Our Sponsors

People in this
Meetup are also in:

Create your own Meetup Group

Get started Learn more
Allison

Meetup has allowed me to meet people I wouldn't have met naturally - they're totally different than me.

Allison, started Women's Adventure Travel

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy