October 3, 2011 7:00 PM - 31 attended

Use of Hadoop for Image Processing in Cloud and Private Data Center Environments

This location is shown only to members

Please RSVP so we could get an accurate headcount for food & drink which will be provided by Idealab.

Rough Agenda:

  1. 7:00pm - 7:15pm Start to Eat & Mingle
  2. 7:15pm - 7:25pm Introductions
  3. 7:25pm - 7:30pm Identify those with open positions (no contingency recruiters)
  4. 7:35pm - 7:37pm Speaker & Talk Introduction
  5. 7:37pm+ The Talk!

 

Here is a synopsis of the talk that George Chang will be giving:

The Lunar Mapping and Modeling Project (LMMP) is tasked to aggregate lunar data, from the Apollo era to the latest instruments on the LRO spacecraft, into a central repository accessible by scientists and the general public. A critical function of this task is to provide users with the best solution for browsing the vast amounts of imagery available. The image files LMMP manages range from a few gigabytes to hundreds of gigabytes in size with new data arriving every day. Despite this ever-increasing amount of data, LMMP must make the data readily available in a timely manner for users to view and analyze. This is accomplished by tiling large images into smaller images using Hadoop, a distributed computing software platform implementation of the MapReduce framework, running on a small cluster of machines locally. Additionally, the software is implemented to use Amazon's Elastic Compute Cloud (EC2) facility. We also developed a hybrid solution to serve images to users by leveraging cloud storage using Amazon's Simple Storage Service (S3) for public data while keeping private information on our own data servers. By using Cloud Computing, we improve upon our local solution by reducing the need to manage our own hardware and computing infrastructure, thereby reducing costs. Further, by using a hybrid of local and cloud storage, we are able to provide data to our users more efficiently and securely. This talk examines the use of a distributed approach with Hadoop to tile images, an approach that provides significant improvements in image processing time, from hours to minutes. This talk describes the constraints imposed on the solution and the resulting techniques developed for the hybrid solution of a customized Hadoop infrastructure over local and cloud resources in managing this ever-growing data set. It examines the performance trade-offs of using the more plentiful resources of the cloud, such as those provided by S3, against the bandwidth limitations s uch use encounters with remote resources. We will outline some of the technologies employed, the reasons for their selection, the resulting performance metrics and the direction the project is headed based upon the demonstrated capabilities thus far.

 

  • Andres Castaņeda
    Andres Castaņeda

    On behalf of Idealab, I would like to thank you all for attending yesterday's meetup. We are excited about getting JPUG back on the map and we look forward to continuing to support the local dev community.

    If any of your missed yesterday meetup, you are welcome to attend out Tech Job Fair to take a look at our space and network with other technologist. Sorry, but no external recruiters allowed. You can view more info at http://jobfair.idealab.com

    All the best,
    Andres Castaņeda

    Posted October 4, 2011 at 10:53 AM
  • You must be a member to post a comment. Join or login.

31 attended
5.00 5.007 (7 ratings)

Idealab

Graciously provided a location for meetings, food, drinks and jobs.

Other nearby
Meetups
Why these groups?
x

The Meetup Groups shown here are topically similar to Pasadena Java User's Group.

Groups are more likely to be displayed here if they:

  • have a Meetup scheduled
  • have a high rating
  • have a group photo
  • are "public" and not "private"
  • have shown they are likely to stick around (older than 30 days)
Find more Meetup Groups
near Pasadena

Log in

  • Not registered with us yet?
or

Log in to Meetup with your Facebook account.

Log in using Facebook

Sign up

or

Join this Meetup Group even quicker with your Facebook account.

Sign up using Facebook
By clicking the "Sign up using Facebook" or "Sign up" buttons above, you agree to Meetup's Terms of Service