R in Big-Data Environments

Our next meeting will be held at Ebay on Thursday February 16. The featured speaker will be Nachum Shacham, Data Scientist at Ebay.

Once again, thank you O'Reilly for sponsoring ouy meeting!

http://photos1.meetupstatic.com/photos/event/d/7/d/e/event_83815262.jpeg

R in Big-Data Environments

Modern massively parallel processing (MPP) platforms like Hadoop and RDBMS, hold Petabytes of structured and semi-structured records that are readily available for analysis. Running R on such platforms enables the analysis of much larger dataset than is possible through the traditional single-server, RAM-based operation. We'll describe our experience in using R to process big data on Teradata Enterprise Data Warehouse and on a large Hadoop cluster. We'll review the challenges of running R in conjunction with these platforms and describe methods for accomplishing this task. The teradataR package that will be described, enables a PC-based R to use the warehouse resource to run statistical functions, e.g., regression, correlation, and parametetric and nonparametric tests on the warehouse-resident tables, while communicating only the much reduced datasets containing the final results. We'll also review our experience in using R to process unstructured log data as well as tabular data on a 1000-node Hadoop cluster. Finally, we'll review a case study, implemented in R, of comparative cost-performance analysis Teradata warehouse and Hadoop.