addressalign-toparrow-leftarrow-rightbackbellblockcalendarcameraccwcheckchevron-downchevron-leftchevron-rightchevron-small-downchevron-small-leftchevron-small-rightchevron-small-upchevron-upcircle-with-checkcircle-with-crosscircle-with-pluscrossdots-three-verticaleditemptyheartexporteye-with-lineeyefacebookfolderfullheartglobegmailgooglegroupshelp-with-circleimageimagesinstagramFill 1linklocation-pinm-swarmSearchmailmessagesminusmoremuplabelShape 3 + Rectangle 1ShapeoutlookpersonJoin Group on CardStartprice-ribbonShapeShapeShapeShapeImported LayersImported LayersImported Layersshieldstartickettrashtriangle-downtriangle-uptwitteruserwarningyahoo

Simple deployment with SIMR and Advanced Shark Analytics with TGFs

Live-Stream Link: http://www.ustream.tv/channel/spark-meetup-feb-5-2014


Ali Ghodsi of Databricks will be presenting 2 talks at the Huawei offices in Santa Clara, CA on Wednesday Feb 5. 


• TGF: Performing advanced analytics in Shark through Table Generating Functions

slides: http://files.meetup.com/3138542/tgf.pptx

This meetup covers two new features, one for Shark and one for Spark. For Shark, we introduce Table Generating Functions (TGFs). These enable users to perform advanced analytics, such as calling ML libraries, from Shark. TGF is a flexible mechanism that lets you wrap existing Spark libraries, supplying them with parameters, and getting results back as tables. The mechanism builds on the new enhanced RDD and SQL table convertors available in Shark.


• SIMR: Seamlessly launching Spark jobs on MapReduce v1 clusters  

For Spark, we now support the ability to launch Spark jobs on Hadoop MapReduce v1 clusters through SIMR (Spark In MapReduce). This deployment mode for Spark is very seamless as it only requires downloading three files and access to an MR1 cluster. SIMR also supports running the Spark REPL inside MR clusters. 


This Meetup will be live streamed and later added to YouTube. 

Join or login to comment.

  • Andy K.

    The video for this meetup is now available on YouTube at http://www.youtube.com/watch?v=5niXiiEX5pE

    February 6, 2014

  • Andy K.

    Thanks to everybody who joined us last night in person at Huawei and via live streaming.

    We'll post an update here when the video recordings of last night's talks are available on YouTube (should be in few days).

    Also, slides for SIMR talk are at http://files.meetup.com/3138542/simr.pptx

    And (reposting for completeness) slides for TGF talk: : http://files.meetup.com/3138542/tgf.pptx

    February 6, 2014

  • A former member
    A former member

    Great, informative!

    February 6, 2014

  • Trung H.

    Hi, I am from London so I can't join the conversation in the next meetup. I just wonder if we can run SIMR for PySpark ? Look forward to your answering.

    February 5, 2014

    • Andy K.

      Hey Trung, if you haven't yet, I recommend you try asking your question on the [masked] mailing list or even email Ali directly.

      February 6, 2014

  • viplav m.

    Thanks for the good talk. Should have mentioned that the live stream would be delayed.. but it was worth the wait :-)

    February 5, 2014

  • Andy K.

    Slides for the TGF talk are at http://files.meetup.com/3138542/tgf.pptx

    1 · February 5, 2014

  • viplav m.

    Technical difficulties for the live stream ? Hope it is being recorded.

    February 5, 2014

  • Joseph W.

    when will the stream start?

    February 5, 2014

  • Scott W.

    February 5, 2014

  • Tim K.

    Looking forward to filming this event !

    1 · January 25, 2014

  • Scott W.

    We will be live-streaming this Meetup. Please only RSVP if you plan on attending in-person.

    January 18, 2014

People in this
Meetup are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy