Maptime: A Geo Platform (part 2 of 2)

In part 1 of this two-part Maptime we focused on installing a number of back end tools that are indispensable for open source geo work: PostgreSQL, PostGIS, Python, and the osm2pgsql script. We looked at some simple queries, the Data Science Toolkit's geocoder, and strategies for acquiring OpenStreetMap data, particularly through the metro excerpts. Most of that work is reviewed in the first 20 or so slides in a still-evolving slide deck. If you didn't attend last week, or even if you did, you should review those slides.

In part 2, we'll hook MapBox's TileMill application up to our OSM database, explore the display layers and shapefiles, and tweak some CartoCSS in order to generate map tiles we'll serve from our own laptops. This will be immensely satisfying after all the grunt work of last week!

We'll also import additional geocoded data, and use Leaflet to lay that data over our tiles. We'll set up a simple web app to deliver that data in GeoJSON format, and, time permitting, we'll look at examples of Leaflet plugins and the power of spatial queries.

I have a few additional download instructions for tonight. As before, they require little attention, but they do take some time. Getting the shapefiles may take 30-40 minutes.


1. If you've never opened TileMill on your computer before, open it, then simply quit. That'll create the file structure you need for the next part.

2. Open your Terminal program and type

cd ~/Documents/MapBox/project

git clone [masked]:erictheise/sfbay_from_osm_deluxe.git

cd sfbay_from_osm_deluxe 

sh get-shapefiles.sh


If you're missing git, you can

brew install git


Looking forward to seeing you all tonight.


Join or login to comment.

  • Spencer B.

    Just wrote a (slightly belated) blog post about the great experience - thanks again!
    http://spencerboucher.com/post/68625161053/map-time-at-stamen

    November 30, 2013

  • Sean K.

    I missed the step to run mb-util to generate tiles locally from our Tilemill mbtiles export. When I try to use leaflet to load TileLayer, the paths that Leaflet expects are different than what I have stored. Can anyone point me in the right direction?

    My computer, zoom level 9 tiles:
    $ ls -R tiles/sfbay/9 [masked]

    tiles/sfbay/9/80:
    197.png 198.png 199.png

    etc. In Leaflet:

    var tilesUrl = 'tiles/sfbay/{z}/{x}/{y}.png',
    tilesLayer = new L.TileLayer(tilesUrl);

    var map = L.mapbox.map('map')
    .setView([-122.4042,37.6795], 9)
    map.addLayer(tilesLayer)


    Leaflet is looking for tiles/sfbay/9/309/510.png, etc.—different scheme than what I have. Any advice?

    November 21, 2013

    • Eric T.

      Sean, tilesUrl needs to hold a proper url, needs to start with 'http://localhost/...;­.

      November 21, 2013

    • Eric T.

      Well, that's assuming you're delivering them via your laptop's httpd server. Even simpler would be to use a 'file' url and point to the proper place on your filesystem, e.g., 'file:///Users/sean­/tiles/sfbay/{z}/{x}/{y}­.png'.

      November 21, 2013

  • Sean K.

    Thanks Eric! Very appreciative of the immense amount of time and effort you put into this. I know it helped me out quite a bit!

    November 20, 2013

  • Rob R.

    Thanks Eric. Great hands-on session with this map stack. Thanks Alan and Stamen for hosting, will be back for more mappiness.

    November 20, 2013

  • Jason F.

    Eric has put in a LOT of time preparing as well as work after the event to document the process for everyone.

    1 · November 20, 2013

  • Toby N.

    For some reason, I didn't have shapeindex installed so I downloaded it from http://support.mapbox.com/discussions/tilemill/339-location-of-shapeindex-command and ran the last command in the shell script manually.

    November 20, 2013

    • Jason F.

      I got around this by installing mapnik (which we may need later on anyway): $ brew update; brew install mapnik Then I re-ran the get-shapefiles script.

      November 20, 2013

    • Jason F.

      FYI - this took 15+ minutes to run on my few-month old, top of the line MacBook Pro... keep that in mind.

      November 20, 2013

  • A former member
    A former member

    im still in the process of catching up and probably won't be able to make it tonight. but thank you so much for the incredible slides Eric, and for Stamen for hosting such a great event!

    1 · November 20, 2013

  • Toby N.

    A few comments on the slide deck:

    1. There's a weird issue on my mac [masked]) where postgres was already installed and that caused things not to work with odd socket errors. I had to move /usr/local/bin to the front of my path to get things to work. I did this by sudo editing /etc/paths and changing the order of the directories.

    2. Slide 19: Typo: createdb has one 'd'

    3. Slide 22: The version of osm2pgsql that brew installed does not support pbf files. I had to download the XML from the teczno site (http://osm-extracted-metros.s3.amazonaws.com/sf-bay-area.osm.bz2) bunzip it and point osm2psql at that.

    4. Slide 22: The database name doesn't match the database created in earlier slides. The command I used is:

    $ osm2pgsql -d sfbay_from_osm sf-bay-area.osm

    November 20, 2013

    • Jason F.

      I have OS X[masked] and my osm2pgsql worked with a PBF file.

      November 20, 2013

    • Jason F.

      $ brew info osm2pgsql
      osm2pgsql: stable[masked]http://wiki.openstree...­
      /usr/local/Cellar/osm2pg­sql/0.82.0 (9 files, 344K) *
      Built from source
      From: https://github.com/mxc...­
      ==> Dependencies
      Build: autoconf ✔, automake ✔, libtool ✔
      Required: geos ✔, proj ✔
      Optional: protobuf-c ✔
      ==> Options
      --with-protobuf-c
      Build with protobuf-c support

      November 20, 2013

  • Rob R.

    I'm getting an error during execution of get-shapefiles script. It seems to get pretty far in downloading and expanding sets then this...

    expanding land-polygons-split-3857...
    processing ne_10m_populated_places...
    Warning 1: One or several characters couldn't be converted correctly from UTF-8 to ISO-8859-1.
    This warning will not be emitted anymore.
    indexing shapefiles
    get-shapefiles.sh: line 54: shapeindex: command not found

    November 20, 2013

    • Jason F.

      I have the exact same issue. shapeindex is no longer included in TileMill... https://github.com/map...­

      November 20, 2013

    • Jason F.

      I solved this issue by $ brew update; brew install mapnik Then I re-ran the get-shapefiles script.

      November 20, 2013

  • Jason F.

    I had an issue when trying to clone the repository from the [masked] URL. I was able to clone using the following command:

    $ git clone https://github.com/erictheise/sfbay_from_osm_deluxe.git

    November 20, 2013

    • Jason F.

      dowloading world_boundaries...
      Warning: Illegal date format for -z, --timecond (and not a file name).
      Warning: Disabling time condition. See curl_getdate(3) for valid date syntax.

      November 20, 2013

    • Eric T.

      Jason, I got those warnings, too, not to worry.

      1 · November 20, 2013

  • Alan M.

    Hi all, I'll be here tonight to help out, but I removed my name from the RSVP list so let more people in off the waiting list. See you tonight!

    November 20, 2013

  • Karsten

    A question on loading the pbf file.
    I started the server by
    $ pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start

    Then I downloaded the sf-bay-area.osm.pbf Then I typed
    osm2pgsql -d maptimesf ~/Downloads/sf-bay-area.osm.pbf Error! Error: Connection to database failed: FATAL: database "maptimesf" does not exist

    I kinda forget what is the database "maptimesf"? How did we build it? Thanks! It is a very useful slides but how can we download the slides to the computer to get access to it anytime?

    November 20, 2013

    • Eric T.

      Aaron, assuming you created the template, you need to create the database from the template, as shown on slide 19 (createdb -T template_postgis sfbay_from_osm). Then you should be able to run osm2pgsql (osm2pgsql -d sfbay_from_osm ~/Downloads/sf-bay-area.­osm.pbf). I may have changed the name from maptimesf in the course of making the slides; it's for the better, and the things we'll do tonight depend on the database being called sfbay_from_osm, so please use that.

      November 20, 2013

  • Luke W.

    Appologies for backing out at the last minute but, unfortunately, work obligations will have me pegged down this evening

    November 20, 2013

  • Rob R.

    Thanks for the slide post Eric! Excellent details. Working to getting caught up and install these warez.

    November 19, 2013

  • Toby N.

    Thanks for the slides, Eric -- they are super useful (and look great)

    -Toby

    November 19, 2013

  • Marina S.

    Can't attend but hope there will be another session soon.

    November 19, 2013

  • Rob R.

    Will this week's meetup be the part 2 of the spatial databases topic or a different topic? If so, I haven't setup the tools yet and need to check out Eric's slides to learn more. Haven't been to maptime yet but have wanted to for a while. Am working on a few map projects currently using different libs/APIs/tools and would be great to meet others and learn more.

    November 18, 2013

  • Amy S.

    Sorry I won't be able to make it this week. I hope there's another round of spatial database fun in the future. :)

    November 17, 2013

Our Sponsors

  • Stamen Design

    Covers meetup dues, provides meeting space and snacks.

People in this
Meetup are also in:

Sometimes the best Meetup Group is the one you start

Get started Learn more
Rafaël

We just grab a coffee and speak French. Some people have been coming every week for months... it creates a kind of warmth to the group.

Rafaël, started French Conversation Group

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy