Hack/Reduce Attempt #1

Please note doors will be open from 7:30pm for nibbles/drinks

 

'Looking at Bristol through Data'

This was the winning topic from the survey. It is a good subject to work upon, I will spend the next few days identifying relevant datasets for us to work with. Any idea's of datasets to make available would be fantastic, so we can pre-load them on to the instances.

Have a location and we have pizza! University of Bristol IT Services R&D / ILRT are sponsoring the space and Armadillo are kindly sponsoring a food fund for the event!

The event will run from Friday at 7:45pm till late. Then will start again on Saturday afternoon/evening with team presentations at 5-6:00pm.

Agenda

Friday

7.30pm drinks and nibbles

7:45-8:00pm - Edward Tufte primer talk (possibly a last minute guest speaker)

8:00 to 8:15pm - Briefing and organise into teams,

8:30 pm to 9:30 pm - Theme planning and selecting datasets, environment setup (gituhub, AWS)

9:30pm - till we get kicked out - more environment, github, AWS, Elastic MapReduce etc... and some coding

 

Saturday

10:00am to 1:00pm - Coding

1:00pm to 1:30pm - Lunch

1:30pm to 5:00pm - More Coding

5:00pm to 6:00pm - 5 minute Presentations and Q&A

6:00 to 6:10pm - Voting and Feedback

(btw hashtag for the event is #SWBigData)

 

The purpose of the event is to create enough interesting data to produce the worlds first South West Big Data infographic, visualisation or live dashboard on a topic we select. All the topics are below, please fill out the survey! The winning team will get south west big data t-shirts!!!

We didn't' manage to find a speaker, so I will be dusting off my keynote and presenting the following primer for the weekend:

- Edward Tufte and the Integrity of Data Visualisation.

Which will be a bit about the methodology behind infographics and visualisations.

Location

The first Hack/Reduce attempt will now be very generously hosted by Simon Price and colleagues at a University of Bristol venue in 8-10 Berkeley Square, BS8 1HH.

Hardware

Using instances and AMI's on Amazon AWS.

Volunteers

If you are interested in volunteering for the event, either as a technical volunteer (if you have a skills such as quickly setting up a Hadoop environment) or just want to help me organise it then please let me know.

Pre-requisites

You will need a computer.

Other than that, there are no pre-required skills. Please just come along and 'have a go' or lurk in the corner and watch. Come as a group or company or come on your own and join in.

There are many disciplines to BigData that don't involve heavy technical lifting, including visualisation with pens, planning and even just having a good knowledge of the subject you are attempting to query can be extremely useful. Often, just a second set of eyes can be very valuable.

Format

The format will stay the same as proposed before with Friday Evening / Saturday Evening group activities and a set of presentations at the end.

Sponsors

University of Bristol (Thanks Simon and colleagues)

Thanks Armadillo!

:)


Paul

Join or login to comment.

  • Simon P.

    Thanks to Duncan Wilkie for publishing my demo map of Bristol's wards on his github account at http://mrdnk.github.com/BristolWardMap/ giving it a more permanent home.

    Hopefully, one day soon we'll get time to integrate this with Duncan's own and Kev Kirkland's historical Bristol ward data visualisations.

    We should also acknowledge the contribution of Dr Peter Wardley, head of the History, Philosophy and Politics Department at UWE, for creating and sharing the Bristol Historical Databases resource all the Hack/Reduce teams used as a starting point.

    4 · February 19, 2013

    • Paul H.

      Yes, I am pretty snowed under for the next few weeks, but I want to bring all of the data together into an interactive d3 visualisation soon.

      February 20, 2013

    • Paul H.

      Yes, a huge thanks to Dr. Peter Wardley, without whom a lot of this data would be lost

      2 · February 20, 2013

  • Kev K.

    Good meetup on Sat. I've posted the IPython notebook for the Life Expectancy in Bristol data and added some comments on the data. The most noticeable point was that there is a 10 year difference between the lowest and highest life expectancies between wards in Bristol.

    http://dataunity.github.com/bristol-open-data/HealthLifeExpectancyByWard.html

    2 · February 19, 2013

    • Paul H.

      Thanks Kev, great to have you there. Great work on the data. Might be good to have a comparison with that data against house prices.

      February 19, 2013

    • Kev K.

      Thanks Paul. Yes, it would be good to take things to the next level and start to look correlations with other open data sets. The charts could use some D3 magic too!

      1 · February 19, 2013

  • Simon P.

    Update on the interactive map of Bristol's districts destined for the Death and Taxes visualisation: Projecting Ordnance Survey's boundary data for Bristol's electoral wards onto OpenStreetMap taught me more than I ever wanted to know about OpenLayers.js and Proj4js. Anyway, the result of more hours than I care to admit now works and is at the temporary home of
    https://fluff.bris.ac.uk/fluff/u2/ecsnp/r9j1wtwEttlNqcuzIUBn3wEIL/
    until it makes it to some online location or gets integrated into our team's demo. Unzip and open map.html. As well as the map demo the raw JSON from the OS is also in the zip.

    3 · February 17, 2013

  • David C.

    This was great! I learnt about new tools, as well as getting to apply other ones in a new context. I am inspired to try so many things, it is hard to pick one to focus on. It is great to see the code and live demos still appearing on github after the event is over. https://github.com/paulharwood/swbd

    2 · February 17, 2013

  • Paul H.

    Thanks to all who attended the First Attempt, great to see people there.

    Big, big thank you to Chris Thurling and Armadillo for the food and drink budget.

    Thanks to Peter Wardley for providing the Historical Datasets and speaking about his 25 year labor of love.

    Plus a huge thanks to Simon Price and his colleagues for arranging and providing the space, rather lovely location, luckily not very far from Pizza Express.

    Some very interesting presentations at the end (if a little morbid).

    Admittedly there is a limit to what can be achieved with a new dataset, but I thought the initial work showed the promise for events like this in the future. Hopefully we can continue to work on the project to bring it up to the standard of current storage/querying formats.

    Thanks again and hope to see you all at the social.

    Cheers

    -- Paul

    (p.s. someone left a Lenovo charger - which is still there at the venue, on Chris's desk I think, and can be picked up on Monday apparently).

    2 · February 17, 2013

  • Duncan W.

    Project wasn't up and quite working at demo time, but here it is ...
    http://deathandtaxes.herokuapp.com/
    - You can click on a Ward to highlight that line.
    - Repo: https://github.com/MRdNk/DeathAndTaxes
    - It's life expectancy data.

    I've put up a heroku app for Bristol Data as JSON as well:
    http://bristol-data.herokuapp.com/
    - Repo: https://github.com/MRdNk/BristolData
    - If you have some JSON files you want to share, I'll take pull requests on the repo, just add the file to the /data/ folder.

    2 · February 16, 2013

  • Roy S

    Ahh! I don't think I can make it tonght :(
    Is it still OK if I go along tomorrow?

    February 15, 2013

    • Paul H.

      Sure, we will be around from 10:00 - 10:30 am :)

      February 16, 2013

    • Roy S

      Sorry I seem to be feeling worse today than yesterday.. Damn I really wanted to attend this one, hope there's another one in the near future..

      1 · February 16, 2013

  • Duncan W.

    At the door, please let me in

    February 16, 2013

  • Chris B.

    Hi All,
    Peter Wardley's "Bristol Historical Resource" CD is now online. Download it here: https://fluff.bris.ac.uk/fluff/u1/cmcpb/qwptChnvPzThPpzxnQ_s_QEGC/
    Link will remain there until: Mon Feb 18 23:59:[masked] UTC

    February 15, 2013

    • A former member
      A former member

      If anyone was curious what's in those "IdeaList" files, I've faffed around a bit and exported them to CSV. Available here https://dl.dropbox.com...­

      February 16, 2013

  • Duncan W.

    Can I come back in please

    February 15, 2013

  • Mandy B.

    Sorry folks. Was really looking forward to this but am stuck having to finish something and i haven't left Cardiff yet, so don't think I will make it there tonight.

    February 15, 2013

    • A former member
      A former member

      I'm just leaving Cardiff now (19.30 to Temple Meads) so I guess I'll arrive about an hour late...

      February 15, 2013

  • David C.

    1. As I live a fair way from Bristol, I am sorry that I won't be able to stay 'until late' on Friday night.
    2. What time will we restart on Saturday?
    3. I will be happy to help people get set up with ssh keys, basic Linux sysadmin for their EC2 instances etc., though I have temporarily forgotten how to open ssh tunnels :)

    February 13, 2013

    • Paul H.

      It really depends on the availability of the UoB people hosting the event. I will know more tomorrow after speaking with them and will update you.

      2 · February 14, 2013

    • Paul H.

      Hi David it looks like a 10 am - 11 am start on Saturday - I will post the itinerary online once I have it confirmed in about an hour or so.

      February 15, 2013

  • Paul M.

    Really wanted to come but now can't make it. Hope it goes well and look forware to hearing about it!

    1 · February 15, 2013

  • Stephen

    I have to work Saturday now, sorry guys.

    February 13, 2013

  • Duncan W.

    BigData n00b, except large Patient Data with SQL. Have some experience with MongoDB. Use Nodejs (Language of choice) Never used Hadoop.

    February 12, 2013

  • A former member
    A former member

    Gutted - can't make it...

    February 11, 2013

  • Steve L.

    will probably be working on some Hadoop internals

    February 11, 2013

  • David C.

    +1 for the Edward Tufte themed talk. I am looking forward to it.

    February 11, 2013

  • A former member
    A former member

    Looking forward to it. Anything I should get ready?

    1 · February 9, 2013

  • A former member
    A former member

    Sorry on holiday.

    February 5, 2013

  • A former member
    A former member

    Just realised it's my birthday weekend! :(

    2 · January 24, 2013

  • David C.

    Limited ability, less experience, but lots of enthusiasm.

    January 23, 2013

19 went

Our Sponsors

People in this
Meetup are also in:

Sometimes the best Meetup Group is the one you start

Get started Learn more
Rafaël

We just grab a coffee and speak French. Some people have been coming every week for months... it creates a kind of warmth to the group.

Rafaël, started French Conversation Group

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy