Building Efficient Web Scrapers: Rust vs. Python for Data Ingestion

Join us for our next Lunch & Learn 💫, where we’ll dive into building efficient web scrapers with Rust and Python. Camille Owens will walk us through the fundamentals of web scraping, key differences from APIs, and best practices for performance, error handling, and production deployment.

Agenda

13:00 - 13:05
Welcome & Introduction
Caroline Morton

13:05 - 13:45
Building Efficient Web Scrapers: Rust vs. Python for Data Ingestion
Camille Owens
Web scraping is a powerful technique for extracting data from websites, but how does it compare to using APIs? In this Lunch and Learn, I will dive into web scraping fundamentals, covering key differences between scrapers and API calls. I will explore Rust and Python libraries like reqwest, scraper, and Polars for data extraction, along with best practices for asynchronous HTTP requests, error handling, and parsing HTML. Will also cover performance trade-offs, cost considerations, and DevOps insights when deploying scrapers in production. Whether you are building data pipelines or exploring new tools, this session will provide valuable insights into efficient data ingestion strategies.