addressalign-toparrow-leftarrow-rightbackbellblockcalendarcameraccwcheckchevron-downchevron-leftchevron-rightchevron-small-downchevron-small-leftchevron-small-rightchevron-small-upchevron-upcircle-with-checkcircle-with-crosscircle-with-pluscontroller-playcrossdots-three-verticaleditemptyheartexporteye-with-lineeyefacebookfolderfullheartglobegmailgooglegroupshelp-with-circleimageimagesinstagramFill 1light-bulblinklocation-pinm-swarmSearchmailmessagesminusmoremuplabelShape 3 + Rectangle 1ShapeoutlookpersonJoin Group on CardStartprice-ribbonprintShapeShapeShapeShapeImported LayersImported LayersImported Layersshieldstartickettrashtriangle-downtriangle-uptwitteruserwarningyahoo

San Francisco Hadoop Users Message Board › Apr 2011 - Oozie

Apr 2011 - Oozie

A former member
Post #: 14
Notes courtesy of Bill Graham:

- About a dozen people at the session
- Only a few were using it in production
- Common challenges include managing the lifecycle of workflow
descriptors that have to be deployed to HDFS
- One possible solution is to automate syncing your SVN descriptor
info with HDFS using a Hudson/Jenkins plugin that does SVN to HDFS
rsync (yuck)
- Other challenges include dependency management of lib/ dir contents
- It was recommended to use ivy/maven repos to manage jar
dependencies like a traditional app and then automate syncing to HDFS
- It was also recommended to make small very generic workflows (like
a single basic sqoop workflow) that can be highly parameterized for
maximum re-use.
- One user (Matt?) uses it quite a bit at his company but has so much
custom code to deal with external dependency management, versioning
and scheduling/triggering, that they're considering switching to
Voldemort.
- Overall consensus was that Oozie seems like a good technology, but
you'll probably need to implement a lot of support code around it for
the issues described.
Powered by mvnForum

Our Sponsors

People in this
Meetup are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy