Skip to content

Details

This Spark Meetup will feature a chance to interact with other Spark users and the developers, as well as two presentations:

Hive on Spark (Shark), the port of Apache Hive to Spark that supports fast in-memory data mining on existing Hive data. How Conviva used Spark to speed up video analytics by 25x: a presentation on Spark applications at Conviva. The meetup will be Tuesday Feb 23rd at Conviva in San Mateo. Food will be available starting at 6:30 PM, and presentations will begin at 7 PM.

More details on the talks:

Shark: Bringing Hive to Interactive Speeds using Spark

(Cliff Engle, Antonio Lupher, and Reynold Xin, UC Berkeley)

Apache Hive offers a powerful platform for large-scale data warehousing, but using Hive for interactive, exploratory data mining is burdensome due to its high latency. We have modified Hive's query planner to allow running queries on Spark, providing an order of magnitude lower latency thanks to Spark's fast execution engine and in-memory caching capabilities. The resulting system, Shark (Hive + Spark), is backwards-compatible with most of the current Hive query language and UDFs, and can work on existing data managed by Hive. We will report on progress implementing Shark and our additions to the Hive language that let users control caching. We are also planning to release a public alpha of Shark soon for interested users.

How Conviva used Spark to Speed Up Video Analytics by 25x

(Dilip Antony Joseph, Conviva)

Conviva monitors and optimizes tens of millions of online video streams daily for premium video brands. We leverage Apache Hive and Spark to analyze video quality, diagnose video delivery problems, detect anomalies, and optimize internal algorithms. Hive has proved extremely useful in querying the large amounts of video data we collect, but is very slow for reports that query the same amount of data multiple times. This is where Spark shines. Using Spark has reduced the time taken to generate some reports from 24 hours to less than 1 hour.

Related topics

You may also like