addressalign-toparrow-leftarrow-rightbackbellblockcalendarcameraccwcheckchevron-downchevron-leftchevron-rightchevron-small-downchevron-small-leftchevron-small-rightchevron-small-upchevron-upcircle-with-checkcircle-with-crosscircle-with-pluscrossdots-three-verticaleditemptyheartexporteye-with-lineeyefacebookfolderfullheartglobegmailgooglegroupshelp-with-circleimageimagesinstagramFill 1linklocation-pinm-swarmSearchmailmessagesminusmoremuplabelShape 3 + Rectangle 1ShapeoutlookpersonJoin Group on CardStartprice-ribbonShapeShapeShapeShapeImported LayersImported LayersImported Layersshieldstartickettrashtriangle-downtriangle-uptwitteruserwarningyahoo

RE: Elastic Filesystems

From: Darren R.
Sent on: Tuesday, January 26, 2010 4:54 PM
Dave,
Good points re additional requirements. ?The varying usage patterns
and storage requirements may actually lead to the conclusion that we
have to roll our own, and won't be able to standardize this type of
infrastructure in many cases - ie Special Purpose Storage Clusters for
different doc types: images, video, documents, etc.

As an example, I know that Facebook built an image server cloud for
this specific use case called Haystack:
http://www.facebo...­

Unfortunately it's not open sourced - but supposedly will be at some point.

Regarding your questions and my current use case:
- frequency of reads vs writes
??A: write once read many, few if any updates

- when you read, do you read the entire file? or just a segment of it?
  A: entire file in the general case, streaming would be a nice bonus

- how "real-time" must replication be
  A: 'slow' is fine

- what's the average file size (giant log file or small docs)
  A: small to 10MB for most, but would like to be able to store logs
as well - unless that requires a 'custom' solution as above

- how scalable do you really need it to be (are you actually building
something that needs petabytes of storage?)
  A: As of now a few terabytes - but elastic is the key here - looking
for the ability to start small but be able to grow linearly thru a
couple of orders of magnitude

Here is a good summary and more complete list of filesystems, worth
reviewing if you are interested in this topic. ?I failed to mention
CloudStore and ZFS in my previous post as well:
https://wiki.musmo...­

Thanks for the links and comments,
Darren

Our Sponsors

  • IT Creations

    A hardware vendor with modern, private meeting spaces for our group!

  • Network Remedy

    IT Management firm providing Marketing, Community Building + food/drink

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy