Monthly Meetup: Using scrapy to Scrape Webpages

We'll be eating pizza, drinking beer and soda, and talking python. Alex will be covering web scraping with the scrapy framework.

 

http://scrapy.org/

Join or login to comment.

  • A former member
    A former member

    very useful, now i am kind of disappointed i didn't use the scrapy library.

    April 18, 2013

  • A former member
    A former member

    One cool thing I found related to Scrapy is a project from the same developers called Scrapely:

    "Scrapely is a library for extracting structured data from HTML pages."
    It's not a crawler, but rather scrapes data from HTML like BeautifulSoup or LXML.
    It uses machine learning algorithms, and you can train a model for a specific domain to grab the specific data you are looking for. Training basically involves pointing it to a specific domain and showing it an example of the data on that page that you want it to capture. It builds its own pattern to extract the save type of information on other pages structured the same way.

    I'll be trying it out today.

    April 18, 2013

    • Alexander P.

      Sounds very interesting. I'm curious if it will work with lists.Also, I found a site today called www.scraperwiki.com that may provide some good information like correct XPath and ways to display the scraped data.

      April 18, 2013

  • Vlad K.

    Same thing. The doors are locked. Can somebody come down?

    April 17, 2013

  • James A K.

    How do you get into this place? The elevator wouldn't take me up and when I went outside , the door locked behind me!

    1 · April 17, 2013

  • Alexander P.

    Hi Everyone! I've decided to focus on the use of Scrapy. Scrapy installation requires a lot of other packages. If you run python on windows without a compiler, it will take about 30 minutes to install. So please try installing it prior to the meetup or start installing it when you arrive.

    April 16, 2013

  • Alexander P.

    This is going to be a great meeting. I will present an intro to web scraping. Bring access to python with you so you can follow along.

    April 4, 2013

    • Alex V.

      What will you be using urllib? BeautifulSoup?

      April 4, 2013

    • Alexander P.

      I'm in the process of reviewing Scrapy; it may be the BeautifulSoup eater:) I havn't finalized the presentation yet.

      April 8, 2013

  • A former member
    A former member

    Hi Everyone,

    Are you or a programmer you know interested in a freelance project involving:

    -Python
    -Java.
    -Putty
    -Bitvise Tunnelier

    If anyone is interested please contact me ASAP.
    See you all the next meetup.

    Jared
    [masked]

    March 26, 2013

18 went

Our Sponsors

People in this
Meetup are also in:

Create your own Meetup Group

Get started Learn more
Allison

Meetup has allowed me to meet people I wouldn't have met naturally - they're totally different than me.

Allison, started Women's Adventure Travel

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy