Billion Tables Project: Large Databases with PostgreSQL

  • November 21, 2013 · 6:30 PM

NYCPUG is going international!  We gladly welcome Alvaro Hernandez Tortosa, CTO of NOSYS and visiting NYC from Spain.  While he is in NYC, Alvaro offered to come present about the "Billion Tables Project" or in other words, how to create very, very, VERY large databases.

The Billion Tables Project explores how to have large PostgreSQL database installations and explores the metadata required to maintain such an installation.  Even while looking at this extreme example, there is a lot of practicality to understanding how PostgreSQL manages its meta-information for your day-to-day operations. In his talk, Alvaro will be exploring:

• The catalogue structure and its storage requirements

• Table creation speed

• Durability tradeoffs to achieve the desired goal

• Strategy to be able to create the 1B tables

• Scripts / programs used.How the database behaves under such a high table count

• Differences in table creation speed and other shortcuts between different PostgreSQL versions

• How the storage media and database memory affects the table creation speed and the feasibility of the task

• If it makes sense to have such a database

Our meetup space will be at Yodle's new office on the 5th Floor of 50 W 23rd St.  We look forward to seeing you there!

Join or login to comment.

  • Mehboob A.

    Very cool "lets push the edges" project.. I should try this at work.. :D

    November 21, 2013

  • A former member
    A former member

    great presentation. I admire anyone who pushes the limits just to see what happens.
    Can someone upload the presentation.

    Thanks,

    November 21, 2013

  • Janet Y.

    Just leaving work...Hope to be able to make it next time.

    November 21, 2013

    • Evan

      Just getting started at 6:55 if you want to head over.

      November 21, 2013

  • ioulian m.

    Stuck at work... Bummed out - was looking forward to it.

    November 21, 2013

  • Konstantin S.

    Can't make it :(

    November 21, 2013

  • Jonathan S. K.

    There will be pizza + soft drinks - looking forward to seeing everyone tonight.

    November 21, 2013

  • Muhammad

    Are there plans to record this presentation and/or post the slides (if any)? I think that would be invaluable for both attendees who would need it as a reference point and reminder as well as for those who may not be able to make it to the meetup. Any plans?

    November 18, 2013

    • Jonathan S. K.

      (Sorry, have to make two replies because I just learned if I hit enter it automatically posts the comment). Generally we have not recorded the meetups in the past due to the cost/setup of doing the recording properly, but we have always tried to make sure that the speakers post their slides.

      November 19, 2013

    • Muhammad

      Thank you. :)

      November 20, 2013

  • Micah L.

    looking forward to this!

    November 19, 2013

  • Mehboob A.

    We have a MySQL database with 500,000 columns.. I wish I could say more..

    November 18, 2013

  • Anson A.

    Prior engagement.

    November 13, 2013

  • Robert T.

    I hope he is using zfs :-)

    1 · November 10, 2013

  • Jonathan S. K.

    From the speaker:

    Alvaro Hernandez T ‏@ahachete
    What do you think, @nycpug 'ers, is it possible to create 1,000,000,000 tables within a single #PostgreSQL database? Tweet ur bets!

    https://twitter.com/ahachete/status/398537892081188864

    1 · November 7, 2013

  • Moshe B.

    cool, looking forward

    November 7, 2013

  • Raffi

    this looks sweet!

    November 7, 2013

Our Sponsors

People in this
Meetup are also in:

Create your own Meetup Group

Get started Learn more
Katie

I'm surpris ed by the level of growth I've seen since becoming an organizer, it's given me more confidence in my abilities.

Katie, started NYC ICO

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy