Re: [betaNYC] Day care map project

From: Anita S.
Sent on: Saturday, March 29, 2014 9:56 AM
Thanks!

> On Mar 29, 2014, at 9:16 AM, Baris Canatan <[address removed]> wrote:
> 
> Hi,
> 
> I made a first update to my original (PHP) scraper. The data is now saved into a mysql db (no duplicates anymore, so no need to run the removeduplicates python script) and introduced some error checking.
> 
> Best,
> Baris
> 
> Quoting Anita Schmid <[address removed]>:
> 
>> Hi all,
>> 
>> Here is an update on the day care map project and a shout out to all python
>> developers (and php or javascript developers):
>> 
>> I contacted the people behind another Bigapp 2013 project that Will pointed
>> out to me (thanks!):
>> http://www.datana...­
>> 
>> They did exactly what I wanted to do, but only once last May and the data
>> has been frozen since. Turns out they had a developer who wrote the scraper
>> in php, javascript etc., but he seems reluctant to share the code. He also
>> says that one cannot automate it and that one would need to write the code
>> in python (this was all relayed to me by the project manager, not the
>> developer).
>> 
>> So, I am back to my initial plan to write scraping code in python and am
>> looking for anybody with experience in python or anybody with experience
>> with php, javascript etc who would be willing to help me figure out how to
>> do this in python (I need to understand how that particular webpage and
>> database works). We have a working scraper in php (thanks Baris) for the
>> highest level of data (names, addresses), so we have a starting point.
>> Also, I have been playing around with the scrapy package (http://scrapy.org...­)
>> and have had some success retrieving data from the database (
>> https://a816-healt...­), but have not
>> succeeded in automating it.
>> 
>> The project is still on github:
>> https://github.com...­ the data with have
>> gathered so far and Sonya is working on geocoding
>> the addresses.
>> 
>> I also contacted the Commissioner for the Department of Health and Mental
>> Hygiene (Mary Travis Bassett) over their online form and asked if they
>> consider releasing the data through NYC Open Data but have not heard back.
>> 
>> Also, everyone who has a NYC Open Data login, please bump the request to
>> release this data here: https://nycopendat...­
>> 
>> Email me at [address removed] if you are interested to get involved!
>> 
>> Thanks, Anita
>> 
>> 
>> 
>> 
>> 
>> 
>> ---------- Forwarded message ----------
>> From: Colegrove, Will (ManhattanBP) <[address removed]>
>> Date: Thu, Mar 20, 2014 at 5:37 PM
>> Subject: RE: [betaNYC] Day care map project
>> To: "[address removed]" <[address removed]>
>> 
>> 
>> Anita:
>> 
>> 
>> 
>> 2 previous bigapps contestants have done similar projects that you might
>> want to look at before starting the scraping:
>> 
>> 
>> 
>> http://2013.nycbi...­
>> 
>> http://2013.nycbi...­
>> 
>> 
>> 
>> -Will
>> 
>> 
>> 
>> William Colegrove
>> 
>> Director of Budget & Transparency
>> 
>> Manhattan Borough President Gale A. Brewer
>> 
>> 1 Centre Street, 19th Fl South
>> 
>> New York, NY 10007
>> 
>> p:[masked]
>> 
>> f:[masked]
>> 
>> [address removed]
>> 
>> 
>> 
>> 
>> 
>> *From:* [address removed] [mailto:[address removed]] *On Behalf
>> Of *Ben Sacks
>> *Sent:* Thursday, March 20,[masked]:34 PM
>> *To:* [address removed]
>> *Subject:* Re: [betaNYC] Day care map project
>> 
>> 
>> 
>> Whoa! I don't have any experience with scraping but I am looking for day
>> care for my child right now and will be super psyche to use this.
>> 
>> 
>> 
>> The one thing I can say is that I was under the impression that day care
>> centers were licensed through the state and not the city, which is likely
>> why it's not in nyc open data.
>> 
>> 
>> 
>> Good luck and please keep me in the loop about progress. Thanks so much for
>> doing this.
>> 
>> Sent from my iPhone
>> 
>> 
>> On Mar 20, 2014, at 5:30 PM, Anita Schmid <[address removed]> wrote:
>> 
>>  Hi all,
>> 
>> 
>> 
>> I would like to make a map of all licensed daycare centers in NYC. The data
>> is online on https://a816-healt...­ but
>> it is not on NYC Open Data despite someone suggesting it a year ago.
>> 
>> 
>> 
>> So, I am thinking of scraping the data off the site above. Anybody
>> interested in working on this with me? Anybody have experience with
>> scraping? I am looking at Scrapy (http://scrapy.org...­), but don't know if
>> that will work.
>> 
>> 
>> 
>> If you are interested or have feedback, please email me at
>> [address removed].
>> 
>> 
>> 
>> Thanks! Anita
>> 
>> 
>> 
>> 
>> 
>> --
>> Please Note: If you hit "*REPLY*", your message will be sent to
>> *everyone*on this mailing list (
>> [address removed])
>> This message was sent by Anita Schmid ([address removed]) from #betaNYC, a
>> Code for America Brigade for NYC <http://www.meetup...;­.
>> To learn more about Anita Schmid, visit his/her member
>> profile<http://www.meetup...;­
>> To report this message or block the sender, please click
>> here<http://www.meetup...;­
>> Set my mailing list to email me As they are
>> sent<http://www.meetup...;­| In
>> one daily email <http://www.meetup...;­ | Don't
>> send me mailing list messages<http://www.meetup...;­
>> 
>> Meetup, POB 4668 #37895 NY NY USA 10163 | [address removed] [image: Image
>> removed by sender.]
>> 
>> 
>> 
>> 
>> 
>> --
>> Please Note: If you hit "*REPLY*", your message will be sent to
>> *everyone*on this mailing list (
>> [address removed])
>> This message was sent by Ben Sacks ([address removed]) from #betaNYC, a
>> Code for America Brigade for NYC <http://www.meetup...;­.
>> To learn more about Ben Sacks, visit his/her member
>> profile<http://www.meetup...;­
>> To report this message or block the sender, please click
>> here<http://www.meetup...;­
>> Set my mailing list to email me As they are
>> sent<http://www.meetup...;­| In
>> one daily email <http://www.meetup...;­ | Don't
>> send me mailing list messages<http://www.meetup...;­
>> 
>> Meetup, POB 4668 #37895 NY NY USA 10163 | [address removed] [image: Image
>> removed by sender.]
>> 
> 
> 
> 
> 
> 
> --
> Please Note: If you hit "REPLY", your message will be sent to everyone on this mailing list ([address removed])
> http://www.meetup...­
> This message was sent by Baris Canatan ([address removed]) from #betaNYC, a Code for America Brigade for NYC.
> To learn more about Baris Canatan, visit his/her member profile: http://www.meetup...­
> Set my mailing list to email me
> 
> As they are sent
> http://www.meetup...­
> 
> In one daily email
> http://www.meetup...­
> 
> Don't send me mailing list messages
> http://www.meetup...­
> Meetup, POB 4668 #37895 NY NY USA 10163 | [address removed]
> 

People in this
Meetup are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy