Online Scraping and Indexing

Day change! Biweekly on Wednesdays.
No longer hybrid! Online only.

Prior to your first session,
please read through "Background" and "Prepare"
on https://base.assembled.app/code/reap#reap,
so you'll be ready to run code which can command Firefox.

Scraping online pages can open many doors;
you can begin building up large databases of public records,
such as research papers, legislation, news articles, or e-commerce collections.

Our group is scraping public records describing grant opportunities:

• https://www.grants.gov/web/grants/search-grants.html
• https://www.grantwatch.com/grant-search.php?search=advanced
• https://grantgopher.com/#/search?pageNumber=1&pageSize=25

Be ready to pull, parse, and index these sources - including a dig through a number of domains to reach the description pages. Our hope is to search through these grants to more easily apply to funds for public programs.

You can see sample scraper code included in the codebase. I can run through these examples during our sessions, though please bring more ideas of pages you'd like to help scrape.