addressalign-toparrow-leftarrow-rightbackbellblockcalendarcameraccwcheckchevron-downchevron-leftchevron-rightchevron-small-downchevron-small-leftchevron-small-rightchevron-small-upchevron-upcircle-with-checkcircle-with-crosscircle-with-pluscontroller-playcrossdots-three-verticaleditemptyheartexporteye-with-lineeyefacebookfolderfullheartglobegmailgooglegroupshelp-with-circleimageimagesinstagramFill 1light-bulblinklocation-pinm-swarmSearchmailmessagesminusmoremuplabelShape 3 + Rectangle 1ShapeoutlookpersonJoin Group on CardStartprice-ribbonprintShapeShapeShapeShapeImported LayersImported LayersImported Layersshieldstartickettrashtriangle-downtriangle-uptwitteruserwarningyahoo

Re: [LACloud-Computing] RE: Elastic Filesystems

From: user 2.
Sent on: Tuesday, January 26, 2010 7:24 PM
I'd definitely suggest looking at MogileFS. ?It's not FUSE-enabled, but it's designed for serving "small" files (e.g., docs, images, etc) and to scale massively

I'm happy to meet to discuss it more and share my own experiences with it.

Dave Viner

On Tue, Jan 26, 2010 at 4:54 PM, Darren <[address removed]> wrote:
Dave,
Good points re additional requirements. ?The varying usage patterns
and storage requirements may actually lead to the conclusion that we
have to roll our own, and won't be able to standardize this type of
infrastructure in many cases - ie Special Purpose Storage Clusters for
different doc types: images, video, documents, etc.

As an example, I know that Facebook built an image server cloud for
this specific use case called Haystack:
http://www.facebook.com/note.php?note_id=76191543919

Unfortunately it's not open sourced - but supposedly will be at some point.

Regarding your questions and my current use case:
- frequency of reads vs writes
??A: write once read many, few if any updates

- when you read, do you read the entire file? or just a segment of it?
?A: entire file in the general case, streaming would be a nice bonus

- how "real-time" must replication be
?A: 'slow' is fine

- what's the average file size (giant log file or small docs)
?A: small to 10MB for most, but would like to be able to store logs
as well - unless that requires a 'custom' solution as above

- how scalable do you really need it to be (are you actually building
something that needs petabytes of storage?)
?A: As of now a few terabytes - but elastic is the key here - looking
for the ability to start small but be able to grow linearly thru a
couple of orders of magnitude

Here is a good summary and more complete list of filesystems, worth
reviewing if you are interested in this topic. ?I failed to mention
CloudStore and ZFS in my previous post as well:
https://wiki.musmo.com/view/General/FileSystems

Thanks for the links and comments,
Darren



--
Please Note: If you hit "REPLY", your message will be sent to everyone on this mailing list ([address removed])
http://www.cloud.la/
This message was sent by Darren ([address removed]) from LA Cloud Computing.
To learn more about Darren, visit his/her member profile: http://www.cloud.la/members/2564725/
To unsubscribe or to update your mailing list settings, click here: http://www.meetup.com/account/comm/
Meetup, PO Box 4668 #37895 New York, New York[masked] | [address removed]


Our Sponsors

  • SkySilk Cloud Services

    Provides free cloud Apps & Tools for devs and business at skysilk.com.

  • IT Creations

    A hardware vendor with modern, private meeting spaces for our group!

  • Network Remedy

    IT Management firm providing Marketing, Community Building + food/drink!

People in this
Meetup are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy