Re: Fwd: [betaNYC] Day care map project

From: Baris Canatan
Sent on: Saturday, March 29, 2014 9:15 AM
Hi,

I made a first update to my original (PHP) scraper. The data is now  
saved into a mysql db (no duplicates anymore, so no need to run the  
removeduplicates python script) and introduced some error checking.

Best,
Baris

Quoting Anita Schmid <[address removed]>:

> Hi all,
>
> Here is an update on the day care map project and a shout out to all python
> developers (and php or javascript developers):
>
> I contacted the people behind another Bigapp 2013 project that Will pointed
> out to me (thanks!):
> http://www.datana...­
>
> They did exactly what I wanted to do, but only once last May and the data
> has been frozen since. Turns out they had a developer who wrote the scraper
> in php, javascript etc., but he seems reluctant to share the code. He also
> says that one cannot automate it and that one would need to write the code
> in python (this was all relayed to me by the project manager, not the
> developer).
>
> So, I am back to my initial plan to write scraping code in python and am
> looking for anybody with experience in python or anybody with experience
> with php, javascript etc who would be willing to help me figure out how to
> do this in python (I need to understand how that particular webpage and
> database works). We have a working scraper in php (thanks Baris) for the
> highest level of data (names, addresses), so we have a starting point.
> Also, I have been playing around with the scrapy package (http://scrapy.org...­)
> and have had some success retrieving data from the database (
> https://a816-healt...­), but have not
> succeeded in automating it.
>
> The project is still on github:
> https://github.com...­ the data with have
> gathered so far and Sonya is working on geocoding
> the addresses.
>
> I also contacted the Commissioner for the Department of Health and Mental
> Hygiene (Mary Travis Bassett) over their online form and asked if they
> consider releasing the data through NYC Open Data but have not heard back.
>
> Also, everyone who has a NYC Open Data login, please bump the request to
> release this data here: https://nycopendat...­
>
> Email me at [address removed] if you are interested to get involved!
>
> Thanks, Anita
>
>
>
>
>
>
> ---------- Forwarded message ----------
> From: Colegrove, Will (ManhattanBP) <[address removed]>
> Date: Thu, Mar 20, 2014 at 5:37 PM
> Subject: RE: [betaNYC] Day care map project
> To: "[address removed]" <[address removed]>
>
>
>  Anita:
>
>
>
> 2 previous bigapps contestants have done similar projects that you might
> want to look at before starting the scraping:
>
>
>
> http://2013.nycbi...­
>
> http://2013.nycbi...­
>
>
>
> -Will
>
>
>
> William Colegrove
>
> Director of Budget & Transparency
>
> Manhattan Borough President Gale A. Brewer
>
> 1 Centre Street, 19th Fl South
>
> New York, NY 10007
>
> p: [masked]
>
> f: [masked]
>
> [address removed]
>
>
>
>
>
> *From:* [address removed] [mailto:[address removed]] *On Behalf
> Of *Ben Sacks
> *Sent:* Thursday, March 20, 2014 5:34 PM
> *To:* [address removed]
> *Subject:* Re: [betaNYC] Day care map project
>
>
>
> Whoa! I don't have any experience with scraping but I am looking for day
> care for my child right now and will be super psyche to use this.
>
>
>
> The one thing I can say is that I was under the impression that day care
> centers were licensed through the state and not the city, which is likely
> why it's not in nyc open data.
>
>
>
> Good luck and please keep me in the loop about progress. Thanks so much for
> doing this.
>
> Sent from my iPhone
>
>
> On Mar 20, 2014, at 5:30 PM, Anita Schmid <[address removed]> wrote:
>
>   Hi all,
>
>
>
> I would like to make a map of all licensed daycare centers in NYC. The data
> is online on https://a816-healt...­ but
> it is not on NYC Open Data despite someone suggesting it a year ago.
>
>
>
> So, I am thinking of scraping the data off the site above. Anybody
> interested in working on this with me? Anybody have experience with
> scraping? I am looking at Scrapy (http://scrapy.org...­), but don't know if
> that will work.
>
>
>
> If you are interested or have feedback, please email me at
> [address removed].
>
>
>
> Thanks! Anita
>
>
>
>
>
> --
> Please Note: If you hit "*REPLY*", your message will be sent to
> *everyone*on this mailing list (
> [address removed])
> This message was sent by Anita Schmid ([address removed]) from #betaNYC, a
> Code for America Brigade for NYC <http://www.meetup...;­.
> To learn more about Anita Schmid, visit his/her member
> profile<http://www.meetup...;­
> To report this message or block the sender, please click
> here<http://www.meetup...;­
> Set my mailing list to email me As they are
> sent<http://www.meetup...;­| In
> one daily email <http://www.meetup...;­ | Don't
> send me mailing list  
> messages<http://www.meetup...;­
>
> Meetup, POB 4668 #37895 NY NY USA 10163 | [address removed] [image: Image
> removed by sender.]
>
>
>
>
>
> --
> Please Note: If you hit "*REPLY*", your message will be sent to
> *everyone*on this mailing list (
> [address removed])
> This message was sent by Ben Sacks ([address removed]) from #betaNYC, a
> Code for America Brigade for NYC <http://www.meetup...;­.
> To learn more about Ben Sacks, visit his/her member
> profile<http://www.meetup...;­
> To report this message or block the sender, please click
> here<http://www.meetup...;­
> Set my mailing list to email me As they are
> sent<http://www.meetup...;­| In
> one daily email <http://www.meetup...;­ | Don't
> send me mailing list  
> messages<http://www.meetup...;­
>
> Meetup, POB 4668 #37895 NY NY USA 10163 | [address removed] [image: Image
> removed by sender.]
>


Our Sponsors

People in this
Meetup are also in:

Log in

Not registered with us yet?

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy