Hi,

I made a first update to my original (PHP) scraper. The data is now  
saved into a mysql db (no duplicates anymore, so no need to run the  
removeduplicates python script) and introduced some error checking.

Best,
Baris

Quoting Anita Schmid <[address removed]>:

> Hi all,
>
> Here is an update on the day care map project and a shout out to all python
> developers (and php or javascript developers):
>
> I contacted the people behind another Bigapp 2013 project that Will pointed
> out to me (thanks!):
> https://www.datana...­
>
> They did exactly what I wanted to do, but only once last May and the data
> has been frozen since. Turns out they had a developer who wrote the scraper
> in php, javascript etc., but he seems reluctant to share the code. He also
> says that one cannot automate it and that one would need to write the code
> in python (this was all relayed to me by the project manager, not the
> developer).
>
> So, I am back to my initial plan to write scraping code in python and am
> looking for anybody with experience in python or anybody with experience
> with php, javascript etc who would be willing to help me figure out how to
> do this in python (I need to understand how that particular webpage and
> database works). We have a working scraper in php (thanks Baris) for the
> highest level of data (names, addresses), so we have a starting point.
> Also, I have been playing around with the scrapy package (https://scrapy.org...­)
> and have had some success retrieving data from the database (
> https://a816-healt...­), but have not
> succeeded in automating it.
>
> The project is still on github:
> https://github.com...­ the data with have
> gathered so far and Sonya is working on geocoding
> the addresses.
>
> I also contacted the Commissioner for the Department of Health and Mental
> Hygiene (Mary Travis Bassett) over their online form and asked if they
> consider releasing the data through NYC Open Data but have not heard back.
>
> Also, everyone who has a NYC Open Data login, please bump the request to
> release this data here: https://nycopendat...­
>
> Email me at [address removed] if you are interested to get involved!
>
> Thanks, Anita
>
>
>
>
>
>
> ---------- Forwarded message ----------
> From: Colegrove, Will (ManhattanBP) <[address removed]>
> Date: Thu, Mar 20, 2014 at 5:37 PM
> Subject: RE: [betaNYC] Day care map project
> To: "[address removed]" <[address removed]>
>
>
>  Anita:
>
>
>
> 2 previous bigapps contestants have done similar projects that you might
> want to look at before starting the scraping:
>
>
>
> https://2013.nycbi...­
>
> https://2013.nycbi...­
>
>
>
> -Will
>
>
>
> William Colegrove
>
> Director of Budget & Transparency
>
> Manhattan Borough President Gale A. Brewer
>
> 1 Centre Street, 19th Fl South
>
> New York, NY 10007
>
> p:[masked]
>
> f:[masked]
>
> [address removed]
>
>
>
>
>
> *From:* [address removed] [mailto:[address removed]] *On Behalf
> Of *Ben Sacks
> *Sent:* Thursday, March 20,[masked]:34 PM
> *To:* [address removed]
> *Subject:* Re: [betaNYC] Day care map project
>
>
>
> Whoa! I don't have any experience with scraping but I am looking for day
> care for my child right now and will be super psyche to use this.
>
>
>
> The one thing I can say is that I was under the impression that day care
> centers were licensed through the state and not the city, which is likely
> why it's not in nyc open data.
>
>
>
> Good luck and please keep me in the loop about progress. Thanks so much for
> doing this.
>
> Sent from my iPhone
>
>
> On Mar 20, 2014, at 5:30 PM, Anita Schmid <[address removed]> wrote:
>
>   Hi all,
>
>
>
> I would like to make a map of all licensed daycare centers in NYC. The data
> is online on https://a816-healt...­ but
> it is not on NYC Open Data despite someone suggesting it a year ago.
>
>
>
> So, I am thinking of scraping the data off the site above. Anybody
> interested in working on this with me? Anybody have experience with
> scraping? I am looking at Scrapy (https://scrapy.org...­), but don't know if
> that will work.
>
>
>
> If you are interested or have feedback, please email me at
> [address removed].
>
>
>
> Thanks! Anita
>
>
>
>
>
> --
> Please Note: If you hit "*REPLY*", your message will be sent to
> *everyone*on this mailing list (
> [address removed])
> This message was sent by Anita Schmid ([address removed]) from #betaNYC, a
> Code for America Brigade for NYC <https://www.meetup...;­.
> To learn more about Anita Schmid, visit his/her member
> profile<https://www.meetup...;­
> To report this message or block the sender, please click
> here<https://www.meetup...;­
> Set my mailing list to email me As they are
> sent<https://www.meetup...;­| In
> one daily email <https://www.meetup...;­ | Don't
> send me mailing list  
> messages<https://www.meetup...;­
>
> Meetup, POB 4668 #37895 NY NY USA 10163 | [address removed] [image: Image
> removed by sender.]
>
>
>
>
>
> --
> Please Note: If you hit "*REPLY*", your message will be sent to
> *everyone*on this mailing list (
> [address removed])
> This message was sent by Ben Sacks ([address removed]) from #betaNYC, a
> Code for America Brigade for NYC <https://www.meetup...;­.
> To learn more about Ben Sacks, visit his/her member
> profile<https://www.meetup...;­
> To report this message or block the sender, please click
> here<https://www.meetup...;­
> Set my mailing list to email me As they are
> sent<https://www.meetup...;­| In
> one daily email <https://www.meetup...;­ | Don't
> send me mailing list  
> messages<https://www.meetup...;­
>
> Meetup, POB 4668 #37895 NY NY USA 10163 | [address removed] [image: Image
> removed by sender.]
>