Building a Python Service Stack

  • January 8, 2014 · 6:15 PM

Title: Building a Python service stack

Abstract: Two years ago at Yelp we broke out our first service from our monolithic Python web application.  Since then we have created over twenty more production services.  In this presentation, I talk about this journey, and in particular, how we have evolved our Python service stack to deal with the challenges that we have encountered in an increasingly distributed production environment.

Speaker Bio: John Billings is Technical Lead of Search Infrastructure at Yelp.  He loves building scalable backend systems and enjoys uninterrupted sleep. Prior to this, he received his PhD from the University of Cambridge by building compilers for Internet routing protocols.

Agenda:

6:15p - Check-in and mingle, with Pizza, Wine and Beer provided by our generous event host, Twitter!

7:00p - Welcome and announcements (Doors close downstairs).

7:10p - Lightning Talks

7:30p - Featured Talk: Building a Python service stack

8:00p - Q & A

8:15p - Second Talk: Brian Wickman talks Twitter's Python distribution format - What is a PEX and why do I care?

8:30p - Second Q & A and more mingling

9:30p - Everybody out!

Lightning talks:

John Sheehan: How embedcurl.com works

Nate Aune: Launcher, a Django-powered service for deploying web applications to Docker containers

If you've encountered something cool recently, please submit a topic for your lightning talk. Please note that recruiting pitches are not acceptable as lightning talks.  Don't know what lightning talks are?  Learn more: http://en.wikipedia.org/wiki/Lightning_Talk

*Please take note of the important check-in details at Twitter*

*Note: Press is not admitted into these events; please e-mail [masked] if you have any questions. Twitter reserves the right to refuse entry to this event. Twitter also reserves the right to remove individuals from Twitter's premises for disruptive or disorderly behavior, or for any other reason in Twitter's sole discretion.

*CHECK-IN PROCESS*

Doors will open at 6:00pm to allow enough time for the check-in process. Please update the name on your account to reflect your FIRST NAME and LAST NAME. Twitter Security and Simeon will be checking IDs at the door. Please enter your full name here. You WILL ONLY be allowed into the event if we have your first and last name (that matches the name on your ID) prior to the event. Since alcohol will be served at the event, we ask that any underage attendees RSVP directly to the meetup host.

We will admit waiting list folks AFTER we admit all confirmed attendees. We are guests at Twitter and we appreciate their generous gifts of pizza, drinks, and beer in addition to their venue. Please be respectful of their space!

Join or login to comment.

  • Simeon F.

    Also Nate Aune's talk on instantly deploying apps with docker and appsembler.com is up at http://youtu.be/SU4WN-FTZDw

    1 · January 27, 2014

  • Simeon F.

    The main talk for this session "Building a Python Service Stack" is up at http://youtu.be/MLM2acV_1yo - thanks Rob and Twitter for the video!

    January 27, 2014

  • Gabriel R.

    Are there links to the presentations and video?

    1 · January 8, 2014

    • Joe S.

      Rob rampaged on one of the lightning talks- "Parsing curl Commands with shlex" is posted at http://youtu.be/QnJ22...­

      1 · January 17, 2014

    • Joe S.

      Someone else should post the others as they're processed next week :)

      January 17, 2014

  • Brian S.

    I enjoyed Brian Wickman's engaging presentation format.

    3 · January 9, 2014

    • Shannon -jj B.

      Hence you can't use an executable.

      January 9, 2014

    • Shannon -jj B.

      Hence you can't use it as an executable.

      January 9, 2014

  • David H B.

    I forgot to ask: do you guys (Yelp) have uwsgi speaking wsgi protocol (with an nginx frontend), or just straight http (with or without an nginx frontend)?

    January 9, 2014

    • Jimmy R.

      For a little context, there is no requirement that one of our services use uWSGI or even Python. Last night John alluded to the fact that we have several Java services. But we want to have a sane default that people use if they don't have unusual requirements and that sane default is currently uWSGI/virtualenv/Python/­Pyramid plus the bits of glue Julian & John talked about. We use the same commands to deploy/start/stop/etc. services regardless of stack. We don't want Ops people to have to know anything about the stack used if they only want to move services around, restart them, add more workers, etc. This is one of the prime motivations for the 'services are self-contained units' abstraction John referred to.

      January 9, 2014

    • David H B.

      Thanks a lot for answering my questions!

      January 9, 2014

  • Matt B.

    Thanks guys - re the main talk, if you give it again, I would give a higher level overview somewhere and dive into the individual bits as they're related to that - how a request flows through the system, how request routing works, what services run where, maybe how it's deployed, etc.

    Also curious if you considered building rpc services using binary protocols - thrift, msgpack, and the like. As well as what sort of threading model you're using in your pyramid workers whether it be standard threads or something like gevent/eventlet/etc since you came from tornado.

    The pex/packaging bit was quite interesting - going to check that out. Please post a link to the longer talk when you can.

    1 · January 8, 2014

  • Joe S.

    Thanks to all the speakers and organizers!

    2 · January 8, 2014

  • Yayoi U.

    Great

    1 · January 8, 2014

  • A former member
    A former member

    Best physics lesson ever! Thank you twitter for hosting and all the speakers.

    4 · January 8, 2014

  • Xing X.

    awesome meetup. waiting for the video recording!

    January 8, 2014

  • Ahmer K.

    Very informative. Lots of great information. Thanks.

    2 · January 8, 2014

  • Howard S.

    Thanks it was fun!!!

    2 · January 8, 2014

  • Eric W.

    Will Twitter have secure bike parking at this meeting?

    4 · January 3, 2014

    • Miraj S.

      can i get a spot to park my bike as well? still space?

      January 8, 2014

    • Kevin

      i am also going to have a bike...

      January 8, 2014

  • Matt B.

    Any ideas what the content of the second talk is?

    January 8, 2014

  • Eamon B.

    I am waitlisted... will I be on twitter's list? Should I bother trying to come?

    January 8, 2014

  • Tim S.

    is it possible to still attend this meetup?

    January 8, 2014

  • Darius P.

    I see 7 spots open; but it says RSVP closed. Any idea why?

    January 8, 2014

    • Hy C.

      That's just a quirk of meetup's website. We closed RSVPs late last night to prepare the guest list for Twitter security. Even after RSVPs are closed, folks can still change their RSVPs to "no" (apparently freeing up space). While no one is currently being moved to "attending" we do expect to have room for much of the wait list. See you tonight.

      January 8, 2014

  • Ahmer K.

    I'll be driving from Palo alto tomorrow. Are there any garages nearby that will be open later than 730p?

    January 7, 2014

    • Simeon F.

      Civic Center garage is your best bet - open till midnight.

      January 7, 2014

    • Ahmer K.

      Thanks!

      January 7, 2014

  • Josh E.

    Excited for this, looking forward to seeing everyone there

    January 7, 2014

  • Giles B.

    Oh waitlist. I was wassail thee.

    January 7, 2014

  • Wade

    CEO of an IT Consulting company that provides one on one introductions to VC`s

    January 7, 2014

  • malcolm

    full name: malcolm mcfarland

    January 4, 2014

    • Hy C.

      Thank you for adding your full name in the comments because we forgot to include the link this month. The link to enter your full name is now added to the event description and also below: https://docs.google.co...­

      January 4, 2014

  • A former member
    A former member

    AV guy from BayPiggies, let me know if you need any help recording this event.

    January 3, 2014

    • Simeon F.

      James! I appreciate your work - I used to be in south bay and for some reason still run @baypiggies. It's been great to catch up on some of the baypiggies sessions via video since I haven't been in a year...

      We've got AV from twitter for this event but if you don't mind I'll keep you in mind for future events. Thanks!

      January 4, 2014

    • A former member
      A former member

      Hey Simeon, good to hear! And yes, definitely okay to keep me in mind for future events.

      January 4, 2014

Our Sponsors

  • Yelp

    Providing food, beverages, venue, and a good time!

People in this
Meetup are also in:

Create a Meetup Group and meet new people

Get started Learn more
Henry

I decided to start Reno Motorcycle Riders Group because I wanted to be part of a group of people who enjoyed my passion... I was excited and nervous. Our group has grown by leaps and bounds. I never thought it would be this big.

Henry, started Reno Motorcycle Riders

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy