Name: Spidering Wikipedia Politely In Async Rust
Start: 2025-02-27T18:30:00-08:00
End: 2025-02-27T20:30:00-08:00
Location: Portland State University -- Fourth Avenue Building

[Rescheduled after snow day]

How many pages are reachable from Wikipedia's page on the Rust programming language in two hops? Around 30,000, it turns out, including pages on wheat flour, Welsh orthography, and the zombie apocalypse.

As it turns out, it's super easy to do this exploration using asynchronous Rust code. Wikipedia offers a cute little REST API for querying links, and it's easy to use Serde to generate requests and parse replies. And if you're feeling guilty about flooding a precious public resource with silly API requests, it's also super easy to do rate limiting.

Jim Blandy will show how to wire up Tokio, Reqwest, and Serde to do the spidering, and whip up a mock server for testing using Warp. The techniques shown work nicely for all kinds of REST API scripting, including, say, GitHub.

Please plan to arrive between 6:30 and 7. Due to limitations of the venue, we need to have someone stand outside and let people in, and we'd like them to be able to attend, so the doors will be effectively closed at 7:00, unless you're a PSU student.

Bart Massey

Jim Blandy

PDXRust

Technology

Mozilla

Open Source

Software Development

Computer Programming

Programming Languages

Portland OR

Rust

Rustlang

Spidering Wikipedia Politely In Async Rust

Portland State University -- Fourth Avenue Building

Share

PDXRust

Spidering Wikipedia Politely In Async Rust

PDXRust

Details

Members are also interested in